Run jest for unit tests of modified files only

Run jest for changed files only instead of all tests

Recently in my organization, we were discussing how to write quality unit tests when suddenly the discussion diverted to what if we can save the evaluation time of unit tests on the pipeline and also use the same for the local development as well. So I needed to make a CLI(command-line interface) for local development as well as use the same on CI(continuous-integration). I was given this opportunity and I am glad we were able to reduce the time on CI(Jenkins) by over 3.5 times. I have shared our impactful results at the bottom. So here is my approach to achieving this.

Node packages used:

1. minimist : https://www.npmjs.com/package/minimist

This is a really good package for building a command-line interface. It is primarily used for parsing the arguments given by the user. It saves a lot of code and subsequently makes the code cleaner.

2. chalk : https://www.npmjs.com/package/chalk

This one is optional but again a really good package for making your CLI more user friendly. It enables you to output text in coloured text.

3. shelljs : https://www.npmjs.com/package/shelljs

This one is again optional. I used this in early development but soon realized we don’t need this as exec, spawn can pretty much achieve the same result. I would recommend you not to use this package.

Approach #1: Using jest’s onlyChanged option

My first approach was to use jest’s--onlyChanged option. If you don't know about it, please visit this. It clearly states that it runs unit tests only those files which are modified in the current git repository.

The problem with this option is that once you commit or push your changes, it won’t be able to find changed files so ultimately no unit tests will be evaluated. Even if we automate to evaluate unit tests before committing, firstly it is not ideal and secondly, this won’t work on CI.

Approach #2: Using jest’s findRelatedTests option

Since the previous approach didn’t work, we moved to the next option provided by jest, ie, --findRelatedTests. If you don't know about it, please visit this. Since we can always fetch changed files in a pull request on local and on CI, why not extract changed react files and pass those paths with this flag. And it does work on local development and CI.

The problem with this was a lot of unnecessary code and we soon realized this can be simplified.

Approach #3: Using jest’s changedSince option

This was our final approach. For this, we used jest’s --changedSince option. If you don’t know about it, please visit this. This came in handy as it reduced a lot of code boilerplate code introduced by the previous approach.

Getting base branch name

Since this option required base branch name against which changed files are determined. On CI(Jenkins), you guys would already have set up to get base branch name but for the local development, we need to call Github API and this would expose Github token to the developers which isn’t recommended. We could have easily made an API that fetches Pull Request details but this adds latency. To tackle this, we used the new Github CLI. One just needs to install it and log in. That's it. Following is the reference code:

Get base branch name, using Github CLI

You can see we used gh pr view command to get pull request details. You can check this command on their website.

Fetching base branch on CI

The next hurdle was to fetch the base branch on CI(Jenkins). Before running the command with the changedSince option, you need to fetch it from the remote as it might be the case where the base branch doesn't exist on the machine. One can easily fetch a branch on a local machine, but what about CI. Jenkins restricts to pull and even fetch remote branches. So to overcome this, we first add refs of the base branch and then fetch it. Following is the reference code:

Fetch base branch on CI(Jenkins)

Constructing the command

Use minimist, chalk and construct the command suing them

Since I used spawn to execute the final command, it expects the array of arguments instead of a string. Also, the user can pass -u to update snapshots. I used chalk and made shortcut helper functions to display coloured output.

Execute the command with live output

Initially, I used node’s exec to execute the command but it truncates the output after max buffer size is reached. It is also recommended to use spawn to display large stdout instead of exec . You can read about both of them here and here.

Display live output on the console, use spawn

Since spawn is event-based instead of callback-based, there is no output buffer size limit. Also, our CI detects success or failure or evaluation based on the exit code, so if you don't write,

childProcess.on("exit", code => {    
   console.log("[EXIT CODE]: " + code.toString());    
   process.exit(code);
});

then CI cant detect if it passed or failed. So we need to exit the parent process with the same exit code as of the child process.

Exposed CLI options

A typical command of a user who hasn’t install Github CLI would look something like:

npm run test -- --baseBranch master --targetBranch current-branch

A typical command of a user who has installed Github CLI and wants to update snapshots as well would look like:

npm run test -- -u

Results

Previously it took roughly 6 mins to execute all the unit tests. Even if there was no change in any react file or test case file, it would still execute all the unit tests. But now, it takes nearly 1min 30 secs to execute the unit tests. It resulted in a nearly 3.5 times faster process to evaluate the unit tests.

One can imagine how much time and resources will be saved on Jenkins as it is costly. Also saving developer’s time is really important.

Some things to NOTE:

• If any dependency changes in your project, then it can make your unit tests fail and they need to be updated. Since node_modules are in the .gitignore you can’t determine if packages have changed as all these jest commands work on git repository changes.

We tackled this by referring to this. We checked if any package version has changed, then run all the tests, else run with changedSince option.

• All three jest options introduced earlier works on the principle of changes in the current git repository. Also, it resolves the whole dependency graph to get all the, directly and indirectly, impacted unit test files from the changes in the files. It internally uses jest-changed-files package to determine tests that need to be executed. So you don’t need to worry about if some files can be left to be evaluated.
• Since we use identity-obj-proxy, so we don’t need to worry about changes in the .scss files that might have required to updated our snapshots.

4. There are other useful options that we explored, like

--bail: exit early as soon as first test case breaks
--maxWorkers <number>: we increased this one more that default and     it impacted the total execution times significantly.

For any further queries or questions, comment down below or reach out to me on my mail : amitsingh5198@gmail.com