shuttle

Open source

String extraction, translation and export tools for the 21st century. "Moving strings around so you don't have to"

657 stars0 likes0 views

About

String extraction, translation and export tools for the 21st century. "Moving strings around so you don't have to"

Shuttle: Magic localization dust

Shuttle is a website allowing for the automatic extraction and reintegration of localizable strings in a code base. It also provides an API where articles can be submitted for translation and retrieved. In addition, it provides a workflow for translators and reviewers optimized for the efficient processing of many strings. Finally, for project managers, it provides a dashboard allowing them to view and manage the progress of a localization effort.

Shuttle can be thought of as a Continuous Integration system for translations.

One typical Shuttle workflow is as follows:

An engineer makes a commit to a Project, and marks that commit as requiring localization in Shuttle.
Shuttle scans the commit for any localizable strings using its {Importer}s.
New or modified localizable strings are marked as pending translation. They appear on the translators’ dashboard.
Translators translate all these strings. They then appear on the reviewers’ dashboard.
Reviewers review and approve translations.
Once all translations applying to a commit are approved for all of a Project's required localizations, the Commit is marked as ready.
When the commit is deployed, Shuttle provides a manifest of translated strings that is downloaded as part of the deploy artifact.

Another typical Shuttle workflow is as follows:

An engineer makes an API call using a Project's api_token to submit a new {Article} (this can be an Article, Email, or anything else).
Shuttle parses the {Article}, splits it up into small pieces of strings.
Shuttle determines which strings need translation while optimizing for efficiency and accuracy. These strings appear on the translators’ dashboard.
Translators translate all these strings. They then appear on the reviewers’ dashboard.
Reviewers review and approve translations.
Once all translations applying to a {Article} are approved for all of a Project's required localizations, the {Article} is marked as ready.
An engineer makes an API call to retrieve the translated Articles.

The whole process is extremely parallelizable: while one commit might be pending translation or review, an engineer can make additional commits with new copy, and they will also sit in the queue awaiting translation. Once any commit is fully localized, it is marked as ready for release. This is true for Articles as well, except, Shuttle doesn't keep versions for Articles and any new submission will override the contents of the previous submission.

Shuttle will refuse to deliver a manifest for a commit or a Article that has not been fully translated and reviewed. To prevent such deploys, engineers should add a test to their CI script that ensures that the manifest endpoint does not return 404.

Getting Started

The easiest way to use Shuttle is with Docker. You will need to download and install a Docker environment. If you do not wish to use Docker, you can also install and configure all the necessary dependencies on your development environment.

Using Shuttle with Docker

To get set up, run docker-compose build to generate all the necessary images. You only have to do this once, and each time you make a change to the source code.

To run the generated images, run docker-compose up. Use docker-compose down to stop the running images. To initialize your database and ElasticSearch indexes for the first time, run docker-compose run web bin/docker-setup.

The Shuttle development database is stored as a Docker volume called shuttle_postgres_data. ElasticSearch data is stored as a Docker volume called shuttle_es_data.

Visit http://localhost:3000 and log in with the username [email protected] and the password password123. This user is created by the seed script.

Create and import your first project!

Setting up your development environment without Docker

Developing for Shuttle requires Ruby 2.4.6, PostgreSQL 9.4, Redis, Tidy, Sidekiq Pro ElasticSearch 5.6, and a modern version of libarchive. To run Shuttle for the first time:

Clone this project.
Install Ruby 2.4.6. If you are using RVM, you can do so using the .ruby-version file.
Run brew bundle to install all dependencies available via Homebrew, which are specified in the Brewfile, or install them manually referencing the Brewfile.
Buy Sidekiq Pro and place your private source URL in Gemfile as specified by the Sidekiq Pro documentation.
Create a PostgreSQL user called shuttle, and make it the owner of two PostgreSQL databases, shuttle_development and shuttle_test:
bash
```
createuser shuttle
createdb -O shuttle shuttle_development
createdb -O shuttle shuttle_test
```
You will need to tell Bundler where the libarchive install directory is. If you installed libarchive using Homebrew, you can do this by running
bash
```
bundle config build.libarchive --with-opt-dir=$(brew --prefix libarchive)
```
Likewise, for Rugged, you will need to tell Bundler where the libgit2 install directory is. If you installed libgit2 using Homebrew:
bash
```
bundle config build.rugged --with-opt-dir=$(brew --prefix libgit2)
```
Make sure that PostgreSQL, Redis, and ElasticSearch are running. If you installed them via Homebrew, running brew info [email protected], brew info redis, and brew info elasticsearch will tell you how to run them.
Install the mailcatcher gem, which is used to receive emails sent in development. (This gem is not a part of the Gemfile because it's typically installed as part of a global or system-wide gemset to be used with all projects.)
Optionally, install the foreman gem, which runs all the processes necessary for development.
Install all required gems by running bundle install.
Run rake db:migrate db:seed to migrate and seed the database.
Run RAILS_ENV=test rake db:migrate to setup the test database.
Initialize the ElasticSearch index for development by running rake chewy:reset.
Do the same for the test indexes: RAILS_ENV=test rake chewy:reset
Verify that all specs pass with rspec spec.

Starting the server

To run the development environment, you will need to start the Web server, the mail client, and Sidekiq.

Running the Web server: rails server
Running the mail client: mailcatcher
Running Sidekiq: sidekiq -C config/sidekiq.yml

These processes can be launched together by running foreman start.

Visit http://localhost:3000 and log in with the username [email protected] and the password password123. This user is created by the seed script.

Create and import your first project!

Adding your first project

You are now an admin user on Shuttle. You can click the "Add Project" button to configure your first project. You will need at least read-only access to this project's Git repository. Set up the locale and importing settings as necessary.

Once the project has been added, you can add a commit for it to import strings from. For starters, try entering "HEAD". Once you click "Add", you should see your Sidekiq server output start to fill up with importers processing all the blobs in your HEAD commit. There may be a delay as the repository is checked out to tmp/repos for the first time.

Refresh the Shuttle home page. When you click on your project name, it should expand to show your commit. The progress bar should be orange and indeterminate, indicating that the commit is being processed. Once processing is finished, the Sidekiq log will quiet down and the progress bar should change to an empty (white) bar, indicating that no translations have been made yet.

Click the progress bar to expand the commit and get detailed status information. You should see three numbers. The badged number on the right is the total number of translatable strings found in your project. Inside the progress bar are two numbers separated by a slash. The first number is the number of finished translations across all required locales (should be zero), and the second number is the total number of required translations (should be the number of translatable strings times the number of required locales, minus any keys that are not applicable to certain locales).

The "Import and approve a localization" field allows you to import an existing localization. For example, if your Rails project already has an "fr.yml" file that you want to import into Shuttle (to save your translators the effort of retyping all those translations), you can use this field to do it.

These are the features typically used by users with the "monitor" role. As an admin you can also explore and use the tools used by translators: The translation/review panel, the global search page, and the glossary. Managing other users is an admin-specific feature.

One last important feature that admins have is the ability to visit the "/sidekiq" URL, which lets them monitor and manage Sidekiq workers.

Deploying to production

Shuttle does not come with a deploy script; you should use whatever deploy system you are comfortable. Your deploy script should, in addition to copying the code and starting the Rails server,

stop and restart the Sidekiq workers (the Sidekiq gem has scripts for this), and
install the cron entries (the Whenever gem has scripts for this).

Documentation

Comprehensive documentation is written in YARD- and Markdown-formatted comments throughout the source. To view this documentation as an HTML site, run rake yard.

CoffeeScript libraries are documented using the YARD format as well, but YARD does not as yet recognize them as documentable files. A .codoopts file is included in case you wish to use Codo to generate the CoffeeScript docs, but as of now Codo does not recognize the ERb files, and does not use the full set of Markdown syntax features used in the documentation.

Architecture

Models

Each {Project} has multiple {Commit Commits} or {Article Articles}. When a Commit or Article is created, it is scanned by {Importer Importers} for localizable strings. These strings are represented as {Translation} records. A base Translation is created in the project's base locale, and preapproved, and pending, untranslated Translations are created for each target locale. These families of Translations are grouped under {Key} records, one for each unique key in the Project. The newly created Translations are in the Project's base locale. Future imports reuse the existing Keys if the source copy is unchanged, otherwise generating new Keys and new Translations for the new source copy. {User Users} with the translator role then fill out pending Translations, and reviewers approve them.

When all of a Commit's/Article's Translations in all of a Project's required locales are marked as approved, the Commit/Article is marked as ready. This Commit's/Article's translated copy can then be exported to a manifest file using an {Exporter}, or localized versions of project files can be generated and downloaded using a {Localizer}.

Article keeps an ordered set of {Key Keys} whereas Commit keeps an unordered set, so that the exporter can put the Translations back together in the right order.

Models make extensive use of advanced PostgreSQL features for efficiency and convenience. Cached counters are updated using triggers and rules, foreign key constraints and hooks are enforced at the database level, and validations are backed up by corresponding CHECK triggers. This helps ensure referential and data integrity even in situations where Rails fails, or outside of the Rails stack. See the various migrations to learn more about the triggers, rules, and co