# parallel_tests
Speedup Minitest + RSpec + Turnip + Cucumber + Spinach by running parallel on multiple CPU cores.
ParallelTests splits tests into balanced groups (by number of lines or runtime) and runs each group in a process with its own database.
Setup for Rails
===============
[RailsCasts episode #413 Fast Tests](http://railscasts.com/episodes/413-fast-tests)
### Install
`Gemfile`:
```ruby
gem 'parallel_tests', group: [:development, :test]
```
### Add to `config/database.yml`
ParallelTests uses 1 database per test-process.
| Process number | 1 | 2 | 3 |
| ENV['TEST_ENV_NUMBER'] | '' | '2' | '3' |
```yaml
test:
database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>
```
### Create additional test database(s)
rake parallel:create
rake parallel:create: # if using multi-db setup, would be `secondary` for example
### Copy development schema into all test databases (repeat after migrations)
rake parallel:prepare
### Run!
rake parallel:test # Minitest
rake parallel:spec # RSpec
rake parallel:features # Cucumber
rake parallel:features-spinach # Spinach
rake "parallel:test[1]" --> force 1 CPU --> 86 seconds
rake parallel:test --> got 2 CPUs? --> 47 seconds
rake parallel:test --> got 4 CPUs? --> 26 seconds
...
Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any 'user'-related tests)
rake "parallel:test[^test/unit]" # every test file in test/unit folder
rake "parallel:test[user]" # run users_controller + user_helper + user tests
rake "parallel:test['user|product']" # run user and product related tests
rake "parallel:spec['spec\/(?!features)']" # run RSpec tests except the tests in spec/features
### Example output
2 processes for 210 specs, ~ 105 specs per process
... test output ...
843 examples, 0 failures, 1 pending
Took 29.925333 seconds
### Run an arbitrary task in parallel
```Bash
RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake "parallel:rake[my:custom:task]"
# limited parallelism
rake "parallel:rake[my:custom:task,2]"
```
### Run migrations in all test databases (repeat after migrations)
rake parallel:migrate
rake parallel:migrate: # for multi-db setup
### Setup environment from scratch (create db and loads schema, useful for CI)
rake parallel:setup
### Drop all test databases
rake parallel:drop
rake parallel:drop: # for multi-db setup
Running setup or teardown once
===================
```
…
```
Even test group runtimes
========================
Test groups will often run for different times, making the full test run as slow as the slowest group.
**Step 1**: Use these loggers (see below) to record test runtime
**Step 2**: The next test run will use the recorded test runtimes (use `--runtime-log ` if you wrote to a location different from default)
**Step 3**: Automate upload/download of test runtime from your CI system [example](https://github.com/grosser/parallel_rails_example/blob/master/.github/workflows/test.yml) (chunks need to be combined, an alternative is [amend](https://github.com/grosser/amend))
### RSpec
Rspec: Add to your `.rspec_parallel` (or `.rspec`), but can also be used via `--test-options='--format x'`:
--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log
### Minitest
Add to your `test_helper.rb`:
```ruby
if ENV['RECORD_RUNTIME']
require 'minitest'
require 'parallel_tests/test/runtime_logger'
# ParallelTests::Test::RuntimeLogger.logfile = "tmp/parallel_runtime_test.log" # where to write it
end
```
results will be logged when `RECORD_RUNTIME` is set, so it is not always required or overwritten.
Loggers
=======
RSpec: SummaryLogger
--------------------
Log the test output without the different processes overwriting each other.
Add the following to your `.rspec_parallel` (or `.rspec`), but can also be used via `--test-options='--format x'`:
--format progress
--format ParallelTests::RSpec::SummaryLogger --out tmp/spec_summary.log
RSpec: FailuresLogger
-----------------------
Produce pasteable command-line snippets for each failed example. For example:
```bash
rspec /path/to/my_spec.rb:123 # should do something
```
Add the following to your `.rspec_parallel` (or `.rspec`), but can also be used via `--test-options='--format x'`:
--format progress
--format ParallelTests::RSpec::FailuresLogger --out tmp/failing_specs.log
(Not needed to retry failures, for that pass [--only-failures](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rspec)
RSpec: VerboseLogger
-----------------------
Prints a single line for starting and finishing each example, to see what is currently running in each process.
```
# PID, parallel process number, spec status, example description
[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar
```
Add the following to your `.rspec_parallel` (or `.rspec`), but can also be used via `--test-options='--format x'`:
--format ParallelTests::RSpec::VerboseLogger
Cucumber: FailuresLogger
-----------------------
Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with '@' to rerun failures.
Usage:
cucumber --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log
Or add the formatter to the `parallel:` profile of your `cucumber.yml`:
parallel: --format progress --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log
but can also be used via `--test-options='--format x'`:
Note if your `cucumber.yml` default profile uses `<%= std_opts %>` you may need to insert this as follows `parallel: <%= std_opts %> --format progress...`
To rerun failures:
cucumber @tmp/cucumber_failures.log
Setup for non-rails
===================
gem install parallel_tests
# go to your project dir
parallel_test
parallel_rspec
parallel_cucumber
parallel_spinach
- use `ENV['TEST_ENV_NUMBER']` inside your tests to select separate db/memcache/etc. (docker compose: expose it)
- Only run a subset of files / folders:
`parallel_test test/bar test/baz/foo_text.rb`
- Pass test-options and files via `--`:
`parallel_rspec -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance`
- Pass in test options, by using the -o flag (wrap everything in quotes):
`parallel_cucumber -n 2 -o '-p foo_profile --tags @only_this_tag or @only_that_tag --format summary'`
Options are:
-n PROCESSES How many processes to use, default: available CPUs
-p, --pattern PATTERN run tests matching this regex pattern
--exclude-pattern PATTERN exclude tests matching this regex pattern
--group-by TYPE group tests by:
found - order of finding files
steps - number of cucumber/spinach steps
scenarios - individual cucumber scenarios
filesize - by size of the file
runtime - info from runtime log
default - runtime when runtime log is filled otherwise filesize
-m, --multiply-processes COUNT use given number as a multiplier of processes to run
-s, --single PATTERN Run all matching files in the same process
-i, --isolate Do not run any other tests in the group used by --single(-s)
--isolate-n PROCESSES Use 'isolate' singles with number of processes, default: 1
--highest-exit-status Exit with the highest exit status provided by test run(s)
--failure-exit-code INT Specify the exit code to use when tests fail
--specify-groups SPECS Specify multiple specs running in multiple processes in a given formation.
Commas separates specs in the same process, pipes separate processes.
Spec not mentioned are run in a separate process.
With '-' the value is read from STDIN.
Cannot use with --single, --isolate, or --isolate-n.
parallel_tests -n 3 . --specify-groups '1_spec.rb,2_spec.rb|3_spec.rb'
Process 1 = 1_spec.rb + 2_spec.rb, Process 2 = 3_spec.rb, Process 3 = remainder
--only-group GROUP_INDEX[,GROUP_INDEX]
Only run the given group numbers. Changes `--group-by` default to 'filesize'.
--only-group-continuous-test-env
Instead of resetting `ENV['TEST_ENV_NUMBER']` when using `--only-group`, the env matches the group index`.
Use when running in parallel with shared resources.
Requires `--only-group`.
-e, --exec COMMAND execute COMMAND in parallel and with ENV['TEST_ENV_NUMBER']
--exec-args COMMAND execute COMMAND in parallel with test files as arguments, for example:
$ parallel_tests --exec-args echo
> echo spec/a_spec.rb spec/b_spec.rb
-o, --test-options 'OPTIONS' execute test commands with those options
-t, --type TYPE test(default) / rspec / cucumber / spinach
--suffix PATTERN override built in test file pattern (should match suffix):
'_spec.rb$' - matches rspec files
'_(test|spec).rb$' - matches test or spec files
--serialize-stdout Serialize stdout output, nothing will be written until everything is done
--prefix-output-with-test-env-number
Prefixes test env number to the output when not using --serialize-stdout
--combine-stderr Combine stderr into stdout, useful in conjunction with --serialize-stdout
--non-parallel execute same commands but do not in parallel, needs --exec
--no-symlinks Do not traverse symbolic links to find test files
--ignore-tags PATTERN When counting steps ignore scenarios with tags that match this pattern
--nice execute test commands with low priority.
--runtime-log PATH Location of previously recorded test runtimes
--allowed-missing COUNT Allowed percentage of missing runtimes (default = 50)
--allow-duplicates When detecting files to run, allow duplicates
--unknown-runtime SECONDS Use given number as unknown runtime (otherwise use average time)
--first-is-1 Use "1" as TEST_ENV_NUMBER to not reuse the default test environment
--fail-fast Stop all groups when one group fails (best used with --test-options '--fail-fast' if supported
--test-file-limit LIMIT Limit to this number of files per test run by batching
(for windows set to ~100 to stay below 8192 max command limit, might have bugs from reusing test-env-number
and summarizing partial results)
--verbose Print debug output
--verbose-command Combines options --verbose-process-command and --verbose-rerun-command
--verbose-process-command Print the command that will be executed by each process before it begins
--verbose-rerun-command After a p