safetest

Safetest: Next Generation UI Testing Library

[![NPM version][npm-image]][npm-url] [![Build status][build-image]][build-url] [![Downloads][downloads-image]][downloads-url]

Safetest is a powerful UI testing library that combines Playwright, Jest/Vitest, and React for a powerful end-to-end testing solution for applications and component testing. With Safetest, you can easily test the functionality and appearance of your application, ensuring that it works as expected and looks great on all devices.

Safetest provides a seamless testing experience by integrating with your existing development environment and offering a familiar, easy-to-use API for creating and managing tests.

Features

• Playwright Integration: Run your tests on real browsers using Playwright. Safetest automatically handles browser management, so you can focus on just writing tests.
  • Screenshot diffing via jest-image-snapshot
  • Video recording
  • Trace Viewer
  • Full control over network layer
  • Powerful overrides for complex test cases
• Jest Integration: Safetest leverages the Jest test runner. Write your tests using familiar Jest syntax and benefit from its powerful assertion library and mocking capabilities.
• Vitest Integration: Safetest can also use the Vitest runner. If you have a vite project you'll probably want to use this
• React Support: Safetest is designed with React applications in mind, so you can easily test your components and their interactions. This allows for focused testing of individual components, for Example testing that <Header admin={true}> behaves as expected.
• Framework agnostic: Safetest also works with other frameworks like Vue, Svelte, and Angular. See the examples folder for more details. Safetest even works to component test a NextJS application
• Easy Setup: Safetest is easy to set up and configure, so you can start writing tests in no time. No need to worry about complex configurations or dependencies; Safetest takes care of it all.
• Easy Auth Hooks: If your app is testing an authenticated application, Safetest provides hooks to handles the auth flow in a reusable method across all your tests.

Why Safetest?

Fundamentally UI tests come in two flavors: Integration test and E2E test.

Integration tests are usually ran via react-testing-library or similar. They are fast, easy to write, and test the components within an application. However they are limited in that they don't actually run the application or render actual items to a screen so regressions like having a bad z-index that causes the submit button to be un-clickable won't be caught by these tests. Another common issue is that while it's easy to write the setup for the test, it's hard to write the events needed to cause things on the page to happen. For example, displaying a fancy <Dropdown /> isn't as simple as just calling fireEvent.click('select') since the js-dom doesn't perfectly match the real browser, so you end up needing to mouseover the label and then click the select. Along the same lines figuring out how to enter text on a smart <Input /> has a similar battle. Figuring out the exact incantation to make this happen is hard and brittle. Debugging why something stopped working is also hard since you can't just open the browser and see what's going on.

E2E tests like Cypress and Playwright are great for testing the actual application. They use a real browser and run against the actual application. They're able to test things like z-index issues, etc. However they lack the ability to test components in isolation, which is why some teams will end up having a Storybook adjacent build to point the E2E test at. Another issue is that it's hard to setup the different test fixtures. For example, if you want to test that an admin user has an edit button on the page while a regular user doesn't, you'll find some ways to override the auth service to return different results. Similarly, component testing isn't possible when we have external service dependencies like OAuth since Cypress and Playwright component testing do not run against an actual instance of the app, so any auth gating can make rendering components impossible.

Essentially we end up with this breakdown:

Integration Tests Pros Integration Test Cons E2E Test Pros E2E Test Cons Easy setup Hard to drive Easy to drive Hard to setup Fast Hard to debug Easy to debug Slow Mock services Hard to override network Easy to override network No service mocking Can't test clickability of element Can test clickability of element Can't test z-index Can test z-index Can't easily set value on <Input /> Easy to set value on <Input /> No screenshot testing Screenshot testing No video recording Video recording No trace viewer Trace viewer No confidence app works E2E Confidence components work Confidence app works E2E No confidence components work

It's almost like the two are complementary, if only there was a way to combine the two...

This is essentially what Safetest is trying to solve. It's a way to combine the best of both worlds. It allows us to write tests that are easy to setup, easy to drive, and can test the components in isolation, while also being able to test the application as a whole, test clickability of elements, and do screenshot testing, video recording, trace viewer, etc..

Consider this example:

// Header.tsx
export const Header = ({ admin }: { admin?: boolean }) => (
  

    

      {admin && 
}
      

    </div>
  </div>
);

…

SafeTest artifacts and reports

SafeTest can output a report of the tests that were run, including a trace viewer of each test, a video of the test execution, and a link to open the tested component in the deployed environment. SafeTest tests SafeTest with SafeTest.

See the reports that is generated.

See the Reporting section for more details.

Getting Started

Note: To quickly try the one of the example app in the project, follow the SafeTest Development section below.

To get started with Safetest, follow these steps:

1. Install Safetest as a dependency in your project:

   npm install --save-dev safetest
   

   Or, if you're using Yarn:

   yarn add --dev safetest
   

   The following instructions assume you're using create-react-app. Look in the examples folder for other setup configurations.

2. Add run command to package.json scripts:

   Add the following line to your package.json scripts section:

   {
     "scripts": {
       "safetest": "cross-env OPT_URL=${TARGET_URL:-http://localhost:3000} react-scripts --inspect test --runInBand --testMatch '**/*.safetest.{j,t}s{,x}' --setupFilesAfterEnv ./setup-safetest.tsx",
       "safetest:ci": "rm -f artifacts.json && OPT_URL=${DEPLOYED_URL} OPT_CI=1 OPT_DOCKER=1 npm run safetest -- --watchAll=false --ci=1 --json --outputFile=results.json",
       "safetest:regenerate-screenshots": "cross-env OPT_DOCKER=1 npm run safetest -- --watchAll=false --update-snapshot"
     }
   }
   

   The preceding script runs the default runner (react-scripts) with a couple of flags and environment variables to make sure Safetest is loaded and run with jest and that all .safetest.tsx test files are tested. You may need to adjust based on your specific setup, e.g., using craco or react-app-rewired instead.

   ---

   If you're using Vitest you'd use these instead:

   {
     "scripts": {
       "safetest": "OPT_URL=${OPT_URL:-http://localhost:3000/} vitest --config vite.safetest.config",
       "safetest:ci": "rm -f artifacts.json && OPT_URL=${DEPLOYED_URL} OPT_CI=1 OPT_DOCKER=1 OPT_ARTIFACTS=artifacts.json npm run safetest -- --run --bail=5",
       "safetest:regenerate-screenshots": "OPT_DOCKER=1 npm run safetest -- --run --update"
     }
   }
   

   ---

   A note about the OPT_URL and similar variables. This is used to pass flags to Safetest which will flow through different testing frameworks spawning threads or any other mechanism that would make command line flag passing practically impossible</s,all>

3. Add setup-safetest.tsx file:

   Create a file called setup-safetest.tsx in the root of your project and add the following code:

   import { setup } from 'safetest/setup';
   
   setup({
     bootstrappedAt: require.resolve('./src/main.tsx'),
   });
   

   This file is the minimal setup required to get Safetest working with your project. It's also where you can configure Safetest by specifying options to the setup function.

   If you're using vitest you'll need to add a vitest.safetest.config.ts file with the following config:

   /// <reference types="vitest" />
   
   import { defineConfig } from 'vite';
   
   // https://vitejs.dev/config/
   export default defineConfig({
     test: {
       globals: true,
       testTimeout: 30000,
       reporters: ['basic', 'json'],
       outputFile: 'results.json',
       setupFiles: ['setup-safetest'],
       include: ['**/*.safetest.?(c|m)[jt]s?(x)'],
       threads: process.env.CI ? true : false,
       inspect: process.env.CI ? false : true,
     },
   });
   

4. Bootstrapping your application

   In order for Safetest to be able to work with your application, you need to bootstrap it to load. This is done by modifying your application's entry point (usually src/index.tsx) as follows:

…

The above magic import makes use of Webpack Context Or Vite Glob import (or whatever flavor dynamic import is available) to bundle the .safetest.tsx files in your project separately. This allows you to write tests for your application in the same project as your application, without having to worry about setting up a separate test project or about the tests being loaded when loading your application in a non-test context. The isDev check is only really needed if you don't want to leak your tests into production, but it's not strictly necessary. In this project it's turned off for the examples since I want to test against the final deployed app to keep things simple.

1. Creating your first tests

   Now that you've set up Safetest, you can start writing your first tests. Create a file called src/App.safetest.tsx and add the following code:

…

1. Running your tests

   <