tachometer

tachometer

tachometer is a tool for running benchmarks in web browsers. It uses repeated sampling and statistics to reliably identify even tiny differences in runtime.

Install | Usage | Why? | Example | Features | Sampling | Measurement modes | Interpreting results | Swap NPM dependencies | JavaScript module imports | Browsers | Performance traces | Remote control | Config file | CLI usage

Install

npm i tachometer

Usage

npx tachometer bench1.html [bench2.html ...]

Why?

Even if you run the same JavaScript, on the same browser, on the same machine, on the same day, you'll still get a different result every time. But if you take enough repeated samples and apply the right statistics, you can reliably identify even tiny differences in runtime.

Example

Let's test two approaches for adding elements to a page. First create two HTML files:

inner.html

append.html

Now run tachometer:

npx tachometer append.html inner.html

Tachometer opens Chrome and loads each HTML file, measuring the time between bench.start() and bench.stop(). It round-robins between the two files, running each at least 50 times.

[==============================================------------] 79/100 chrome append.html

After a few seconds, the results are ready:

…

This tells us that using the document.body.append approach instead of the innerHTML approach would be between 90% and 92% faster on average. The ranges tachometer reports are 95% confidence intervals for the percent change from one benchmark to another. See Interpreting results for more information.

Features

• Measure your own specific timings with the /bench.js module, by setting the window.tachometerResult global (or by polling an arbitrary JS expression), or measure First Contentful Paint on any local or remote URL.

• Compare benchmarks by round-robin between two or more files, URLs, URL query string parameters, or browsers, to measure which is faster or slower, and by how much, with statistical significance.

• Swap dependency versions of any NPM package you depend on, to compare published versions, remote GitHub branches, or local git repos.

• Automatically sample until we have enough precision to answer the question you are asking.

• Remote control browsers running on different machines using remote WebDriver.

Sampling

Minimum sample size

By default, a minimum of 50 samples are taken from each benchmark. You can change the minimum sample size with the --sample-size flag or the sampleSize JSON config option.

Auto sample

After the initial 50 samples, tachometer will continue taking samples until there is a clear statistically significant difference between all benchmarks, for up to 3 minutes.

You can change this duration with the --timeout flag or the timeout JSON config option, measured in minutes. Set --timeout=0 to disable auto sampling entirely. Set --timeout=60 to sample for up to an hour.

Auto sample conditions

You can also configure which statistical conditions tachometer should check for when deciding when to stop auto sampling by configuring auto sample conditions.

To set auto sample conditions from the command-line, use the --auto-sample-conditions flag with a comma-delimited list:

--auto-sample-conditions=0%,10%

To set auto sample conditions from a JSON config file, use the autoSampleConditions property with an array of strings (including if there is only one condition):

{
  "autoSampleConditions": ["0%", "10%"]
}

An auto sample condition can be thought of as a point of interest on the number-line of either absolute milliseconds, or relative percent change. By setting a condition, you are asking tachometer to try to shrink the confidence interval until it is unambiguously placed on one side or the other of that condition.

In the following example, we have set --auto-sample-conditions=10%, meaning we are interested in knowing whether A differs from B by at least 10% in either direction. The sample size automatically increases until the confidence interval is narrow enough to place the estimated difference squarely on one side or the other of both conditions.

           n=50  X -10% X +10%
                        n=100 ✔️ -10% X +10%
                                     n=200 ✔️ -10% ✔️ +10%

  |---------|---------|---------|---------| difference in runtime
-20%      -10%        0       +10%      +20%

n    = sample size
 = confidence interval for percent difference of mean runtimes
✔️    = resolved condition
X    = unresolved condition

In this example, by n=50 we are not sure whether A is faster or slower than B by more than 10%. By n=100 we have ruled out that B is faster than A by more than 10%, but we're still not sure if it's slower by more than 10%. By n=200 we have also ruled out that B is slower than A by more than 10%, so we stop sampling. Note that we still don't know which is absolutely faster, we just know that whatever the difference is, it is neither faster nor slower than 10% (and if we did want to know, we could add 0 to our conditions).

Note that, if the actual difference is very close to a condition, then it is likely that the condition will never be met, and the timeout will expire.

Measurement modes

Tachometer supports four modes of time interval measurements, controlled with the measurement config file property, or the --measure flag.

If measurement is an array, then all of the given measurements will be retrieved from each page load. Each measurement from a page is treated as its own benchmark.

A measurement can specify a name property that will be used to display its results.

Performance API

Retrieve a measure, mark, or paint timing from the performance.getEntriesByName API. Note this mode can only be used with a config file.

For example, in your benchmark:

performance.mark('foo-start');
// Do some work ...
performance.mark('foo-stop');
performance.measure('foo', 'foo-start', 'foo-stop');

And in your config file:

"benchmarks": [
  {
    "measurement": {
      "mode": "performance",
      "entryName": "foo"
    }
  }
]

The following performance entry types are supported:

• measure: Retrieve the duration of a user-defined interval between two marks. Use for measuring the timing of a specific chunk of your code.
• mark: Retrieve the startTime of a user-defined instant. Use for measuring the time between initial page navigation and a specific point in your code.
• paint: Retrieve the startTime of a built-in paint measurement (e.g. first-contentful-paint).

Callback

By default with local (non-URL) benchmarks, or when the --measure flag is set to callback, your page is responsible for calling the start() and stop() functions from the /bench.js module. This mode is appropriate for micro benchmarks, or any other kind of situation where you want full control over the beginning and end times.

Global result

When the --measure flag is set to global, then you can assign an arbitrary millisecond result to the window.tachometerResult global. In this mode, tachometer will poll until it finds a result assigned here.

const start = performance.now();
for (const i = 0; i    Wider confidence interval
                      High variance and/or low sample size

            Narrower confidence interval
                 Low variance and/or high sample size

 |---------|---------|---------|---------|
-1%      -0.5%       0%      +0.5%      +1%

The way tachometer shrinks confidence intervals is by increasing the sample size. The central limit theorem means that, even when we have high variance data, and even when that data is not normally distributed, as we take more and more samples, we'll be able to calculate a more and more precise estimate of the true mean of the data.

Swap NPM dependencies

Tachometer has specialized support for swapping in custom versions of any NPM dependency in your package.json. This can be used to compare the same benchmark against one or more versions of a library it depends on.

Use the benchmarks.packageVersions JSON config property to specify the version to swap in, like this:

{
  "benchmarks": [
    {
      "name": "my-benchmark",
      "url": "my-benchmark.html",
      "packageVersions": {
        "label": "my-label",
        "dependencies": {
          "my-package": "github:MyOrg/my-repo#my-branch"
        }
      }
    }
  ]
}

The version for a dependency can be any of the following:

• Any version range supported by NPM, including semver ranges, git repos, and local paths. See the NPM documentation for more details.

• For monorepos, or other git repos where the package.json is not located at the root of the repository (which is required for NPM's git install function), you can use an advanced git configuration object (schema) in place of the NPM version string, e.g.:

  json复制

  {
    "benchmarks": [
      {
        "name": "my-benchmark",
        "url": "my-benchmark.html",
        "packageVersions": {
          "label": "my-label",
          "dependencies": {
            "my-package": {
              "kind": "git",
              "repo": "[email protected]:MyOrg/my-repo.git",
              "ref": "my-branch",
              "subdir": "packages/my-package",
              "setupCommands": ["npm install", "npm run build"]
            }
          }
        }
      }
    ]
  }

You can also use the --package-version flag to specify a version to swap in from the command-line, with format [label=]package@version. Note that the advanced git install configuration is not supported from the command line:

tach mybench.html \
  [email protected] \
  --package-version=my-label=my-package@github:MyOrg/my-repo#my-branch

When you specify a dependency to swap, the following happens:

1. Th