Test Coverage

We have a few testing methods within the ink ecosystem:

Each uses a different framework so it's important to determine which kind of testing is appropriate for your use case

The ink ecosystem contains several environments:

Our end-to-end and visual regression tests both interact with the Exposé environment

Testing Library is the preferred choice but many tests were written in Jest and Enzyme. Regardless of the library, unit tests are predominantly used to test non-visual component behavior and React hooks.

Tests for each component should live in a <component name>.spec.tsx file within the same folder of the component

Don't reference CSS IDs or class names. CSS identifiers of Ink components are subject to change without warning. Majority of our components are now styled which uses a dynamic and unique identifier. Referring to these identifiers in other code, such as unit tests, makes the other code more fragile.

Each test within a suite should test a single function or component behavior. Use selectors that resemble how users interact with the code, such as labels and content text. Read more here or here.

Button.spec.tsx

describe("Button", () => {
  describe("structure", () => {
    it("should render a button tag", () => {
      const wrapper = mount(<Button />);

      expect(wrapper.find("button").length).toBe(1);
    });
  });

  describe("callbacks", () => {
    it("should call prop on click", () => {
      const props = { onClick: jest.fn() };
      const wrapper = mount(<Button {...props} />);

      wrapper.find("button").simulate("click");

      expect(props.onClick).toHaveBeenCalled();
    });
  });
})

To run the entire test suite:

yarn test

To see the tests coverage as well as the tests results:

yarn test:coverage

To run a specific component/test suite, pass a pattern to the command such as the name of a component or the path to the file you're trying to run:

yarn test <pattern>

We use Cypress

All tests are named as <component name>.cy.js and live within the cypress folder. We use this to replicate expected user behavior and all tests are ran against the Expose environment.

Don't reference CSS IDs or class names. CSS identifiers of Ink components are subject to change without warning. Majority of our components are now styled which uses a dynamic and unique identifier. Referring to these identifiers in other code, such as unit tests, makes the other code more fragile.

Use selectors that resemble how users interact with the code, such as labels and content text

cypress/e2e/newTable.cy.js

Cypress.config("baseUrl", `${Cypress.config().baseUrl}NewTable`);

describe("NewTable", () => {
  it("should be able to select rows", () => {
    const selector = 'table[data-testid="newtable-selectable"]';
    cy.get(selector).find('input[id="123"]').should("be.checked");

    cy.get(selector).find('label[for="123"]').click();
    cy.get(selector).find('input[id="123"]').should("not.be.checked");

    cy.get(selector).find('label[for="234"]').click();
    cy.get(selector).find('input[id="234"]').should("be.checked");
  });
})

Cypress tests run against the Expose, so, we must run Expose:

yarn start:expose

Assuming you are running Expose on port 3000, you'll open a second terminal window to run Cypress:

yarn e2e:local

If you want to run Cypress against a different port, you can modify using:

yarn e2e --config baseUrl="http://localhost:<your-port>/"

The Cypress screen will present you some choices:

Cypress tests are live. If an errors occur, cypress will show where the problem is. It can also automatically re-run tests if you leave it open and modify a component's code. However, this doesn't work with samples are samples need to be scraped so Expose must be restarted.

Note: sometimes tests ran in CircleCI yield different results as CircleCI is running tests in "production" mode

We use BackstopJS to collect sample snapshots from our internal testing environment, Exposé. The goal is to cover as many visual variants in each component as possible through atomic sampling allowing us to check for cascading visual effects.

We are predominantly looking for unexpected changes to the visuals

There are only two ways to generate a snapshot;

Names of each scenario must be unique!

Note: Exposé sample snippets and backstop scenarios (added via .backstop.json) don't conflict as they are namespaced to their respective environment

We use a custom schema to make matching our Exposé sample snippets easy. A script is ran in the background to convert it to Backstop's format.

Let's say our NewTable component has 2 samples, "Base NewTable" and "NewTable with twiddle rows"

NewTable/samples.js

[
  {
    group: "NewTable",
    name: "Base NewTable",
    environments: ["expose"],
    code: `<NewTable/>`,
  },
  {
    group: "NewTable",
    name: "NewTable with twiddle rows",
    environments: ["expose"],
    code: `<NewTable/>`,
  },
]

However, this only snapshots the initial state.

If we want to snapshot additional visual checks for the scenario "NewTable with twiddle rows", we need to add them as follows:

NewTable.backstop.json

[
  {
    // The original sample is named "NewTable with twiddle rows" and
    // we're testing the visual when a single is expanded row

    "sampleName": "NewTable with twiddle rows"
    "state": "Single row expanded",
    "clickSelector": "#twiddle-table tbody > tr"
  },
  {
    // The original sample is also "NewTable with twiddle rows" but this time
    // we're checking the visual when all rows are expanded

    "sampleName": "NewTable with twiddle rows"
    "state": "All rows expanded",
    "clickSelector": "button[data-testid='expand-all']"
  },
]

Unfortunately this task isn't automated so if you update a sample name, you must also update the corresponding backstop scenario names.

Similar to how we can set different environments[] in each sample snippet, we use the key breakpoints[] to include snapshots at various screen sizes.

By default, we snapshot at the desktop size. Additional options include:

breakpoints: ['mobile', 'tablet-portrait', 'tablet-landscape', 'desktop-large', 'ultra-wide'],

Technical note: Backstop uses viewports but to align ourselves between sample snippets and additional scenarios, breakpoints is used in both places

We know that Autograph's "Sign here" flag hides itself in mobile only, so we will add a snapshot to each breakpoint available to ensure that's the case:

Autograph/samples.js

[
  {
    group: "Autograph",
    name: "Default Autograph",
    environments: ["expose"],
    breakpoints: ["mobile", "tablet-portrait", "tablet-landscape", "desktop-large"],
    code: `<Autograph preventAutoFocus name="Tagg Palmer" id="autograph-default" />`,
  },
]

There are only two reasons to use a skip:

We usually need a trigger to open a component like Modal. We don't need to screenshot the trigger itself (in this case a Button), so we can skip the initial state:

Modal/samples.js

[
  {
    group: "Modal",
    name: "Base Modal",
    skipInitialBackstop: true,
    environments: ["expose"],
    code: `() => {
      const [isOpen, setOpen] = React.useState(false);

      return (
        <>
          <Button onClick={() => setOpen(true)}>Open Modal</Button>
          {isOpen && <Modal />}
        </>
      );
    }`,
  },
]

Modal.backstop.json

[
  {
    sampleName: "Base Modal",
    clickSelector: "button",

    // Instead the sample itself, we want the whole screen so we must add:
    selectors: ["viewport"],
  },
]

Here are a few options we commonly use:

[
  {
    sampleName: "Component with dropdown",
    state: "active",

    // Perform one or more clicks before capturing the snapshot
    clickSelectors: ["#button1", "#button2"],
  },
  {
    sampleName: "Component with buttons",

    // Scrolls to the given selector before taking the snapshot
    scrollToSelector: "#component-with-buttons",
  },
  {
    sampleName: "Component that hovers",
    state: "hover",

    // A snapshot of the hover state can be captured by a snapshot
    hoverSelectors: ["#button1", "#button2"],
  },
]

See Backstop's documentation for all available options

We want our snapshots to be consistent which means we have to simulate the test to hold the given state (e.g. hover, active, focus) on capture. Sometimes this requires modification to ignore focus rings or animation delays in order to not produce flaky tests.

We have a few custom exposes to help capture design issues spanning across multiple components such as the Disabled to prevent color inconsistently. To add backstop scenarios, we must first add the Expose manually before adding a corresponding backstop file.

⚠️ Although the idea of Backstop is to test against a reference, we can run a singular copy for sanity checks such as naming issues. The local snapshot doesn't produce the same capture as the one ran in CI.

Must be running Expose first

yarn start:expose

(Optional, in a separate terminal window) To skip rebuilding each time there are sample changes:

yarn scrape watch

In a separate terminal window, run backstop (with desired flags)

yarn backstop:test:local

The reference can actually be any branch but using master as the example:

// This command builds the Exposé for master and serves it at localhost:8000
yarn expose:build && yarn expose:serve

// Create reference snapshots
yarn backstop:reference

After all reference snapshots are created, you can stop the Exposé and switch to the branch you need to compare:

git checkout your_branch_name

// This command will start Exposé again but on your_branch_name this time
yarn expose:build && yarn expose:serve

// Run against the master references we created earlier
yarn backstop:test

Only needs a partial match against the filename which contains:

(using local)

// To query all snapshots from NewTable:
yarn backstop:test:local --filter=newtable

// Run a specific sample named "NewTable with buttons":
yarn backstop:test:local --filter=newtable-with-buttons

// Run all Exposé samples for Alert:
yarn backstop:test:local --filter=alert-expose

// Run all Backstop scenarios:
yarn backstop:test:local --filter=backstop

// Any sample that includes (z-index)
yarn backstop:test:local --filter=z-index

Valid failing tests:

Always check the report (local or on CI) and review each failing test. If you don't know why something are failing, ask for help.

Example of failing a Backstop suite:

Example of a misspelled selector will bring up a badger:

If tests are failing due to intentional changes, you must approve the whole suite (to be used as the next reference) before merging, in your local terminal on your branch locally:

⚠️ Important! Never merge a failing test without checking

yarn backstop approve <FAILING_REPORT_URL_FROM_CI>

# Example url: https://output.circle-artifacts.com/output/job/74206be8-1b50-418b-92d9-36e8c2dbdf37/artifacts/0/backstop_data/html_report/index.html

# Using the above, it would look like this:
yarn backstop approve https://output.circle-artifacts.com/output/job/74206be8-1b50-418b-92d9-36e8c2dbdf37/artifacts/0/backstop_data/html_report/index.html

This generates a commit automatically and you will have to push it up to your branch for Backstop to rerun. It should pass the check. If it doesn't, there's probably a flaky test.

⚠️ It's best to check Slack to see if there's currently a merge going on at #ink-backstop

Backstop flakes were more common before but why not avoid having to re-approve backstop if someone merges a second before you (only applies if their branch updates the Backstop reference).

Sometimes we have have multiple branches that all need a Backstop approval so to avoid merging traffic, send a message with claim_backstop where a bot will declare your intentions. Once the PR is merged, use unclaim_backstop to let others know they can proceed with approving their branch's Backstop.