How We Stopped Checking Side Effects and Learned to Love Snapshot Testing

Testing often involves trying to figure out results from bits and pieces of information like class names and number of elements. In this blog, UI developer Bugra Firat shows how snapshot testing provides a better way to add more robust assertions that look at what exactly is rendered.

In xMatters we have been using React for the past few years, and have strived to create good tests with good coverage without getting in the way of developers.

We’ve found Jest to work very well for us, especially its introduction of the idea of “snapshot testing” which delivers very resilient and fast tests.

In this post, I’ll be focusing on the evolution of how people tested their JavaScript on the front end historically through today. I’ll highlight some methods that have worked well for us at xMatters, both in terms of understanding what it is we’re really interested in testing and avoiding pitfalls that might slip by otherwise unnoticed.

Checking Side Effects
Most unit tests we’ve learned to write in the past amount to what is essentially checking for “side effects” of our components. Often, we might check for the number of elements on a list, assert if a class name is present, or check that the content matches.

Unfortunately, in a codebase that promotes re-use of components and gets contributions from a lot of different developers, such tests can miss breaking changes easily. I will try to illustrate below how even the smallest mistakes can provide false-positives or false-negatives.

We will work off an example Todo App that’s in development. We have ‘Todos,’ a list of ‘Items’ that are rendered on the page. First iteration was well received and has “good” tests.

If you take a look at the tests, everything looks okay and well-tested. Now imagine, the next sprint, a developer comes in and makes just a small change. Are our tests robust enough to catch this failure?

All tests are still passing. Looking more closely, clearly there was a logic error, but our “check-for-side-effects” tests did not catch it. One solution is to write additional tests to check for this case as well. Next sample shows this, but look… I’ve made another error, and the tests are still passing, which is a false-positive.

Do we then add more test cases? Every time we discover new edge cases or potential errors, we need to add more and more assertions to make sure our components are rendering what we expect them to. Is there a better way of adding more robust assertions that look at what exactly is rendered, instead of trying to figure it out from bits and pieces like class names and number of elements?

Snapshots
With the introduction of snapshot testing, the answer to the above question is now affirmative! Snapshot testing is essentially an assertion that what the component renders is the same. No need to check number of elements, or whether something has a class name toggled. We check the rendered component exactly as it appears against our expectation, which again describes the component exactly.

Let’s see the same example above, but with snapshot tests now. Notice how any minor error is immediately caught, because we don’t have the same “testing burden”: we can compare exact snapshots.

We’ve found this kind of “defensive” testing invaluable for our uses. This brings any unintended changes to the forefront, and forces developers to consider the effects of their changes to other components, which might be dependent on the component they are modifying. Having an explicit “update snapshots” step makes the intent clear and provides an additional layer of attention to keep things up to date.

Now for the pros and cons
Before closing off, I would like to provide a list of pros and cons that we’ve seen so far. Obviously, this list is neither comprehensive nor absolute, but it’s satisfied most of our requirements in terms of day-to-day development work with React.

Some of the drawbacks:

• Fatigue: when snapshots are first introduced, developers unfamiliar with them or the need to update them might blindly update snapshots every time they run tests. This is dangerous as an update-snapshots-whatever-the-case approach effectively bypasses the assertions and makes the tests useless.
• Solution: one solution to this is to reinforce the idea that the ‘*.snap.js’ files are code and should be treated as such. They should be checked into version control and read over by the author to see if the markup matches what’s expected and that it makes sense.
• Missing data: sometimes it’s easy to miss a rogue ‘undefined,’ ‘NaN’ or ‘null’ lurking among the snapshots, especially combined with the update-on-the-fly mentality mentioned above.
• Solution: even though it might seem inevitable to run into this case, practicing code review will reduce this possibility as other developers will often catch faulty snapshots.
• Conflict resolution: Especially when a merge conflict is in a ‘*.snap.js’ file, conflict resolution can be a bit painful. We’ve found that in most cases the snapshots need to be carefully regenerated. This can cause tedious work but happens relatively infrequently and hasn’t been a major concern.
• Solution: using atomic, focused pull requests and merging usually avoids the case where the same snapshot file and component are updated/worked on at the same time.
• Huge snapshot files: if you are `mount`ing big components with lots of children, the snapshots can get too big to be useful.
• Solution: Using ‘shallow’ rendering and testing only the component in question eases this issue. When testing a parent component with a very deep children tree, consider only the behavior of the parent, and test the children separately on their own.

Ultimately, the drawbacks we’ve seen serve to highlight the importance of a developer’s attention to code, which is irreplaceable. Snapshots won’t solve anything for careless or lazy approaches, but makes life easier by:

• Introducing a clear pattern for testing components, from the smallest 1-2 liners to more complicated ones, with little overhead and boilerplate testing code.
• Showing explicit confirmation of snapshots that directs developer attention to side-effects and other components, depending on what they’re working on.
• Making it easier to view changes in code: expectations are in the same format as what is being rendered, and this also helps with code reviews.
• Rendering easier diffs for viewing failures. I think we can all agree that the snapshot diff is much friendlier and approachable, even if you’re not a developer. This is a big win in terms of developer experience.

Not perfect, but potentially extremely useful
As useful as we’ve found snapshot testing to be for our purposes, it’s not a silver bullet for all UI tests, and there are differing opinions, some of which you can see here and here. In my opinion, snapshot tests are extremely useful and go with the “defensive testing” idea that forces collaborating developers to ensure their changes are not breaking another piece of the UI. Certainly, it’s a very welcome addition to the unit testing arsenal, and I believe it increases productivity, especially coupled with “watch mode” for fast iteration.

Does snapshot testing strike a chord with your experiences? We’d love to hear how you test and what types of tests provide the best results! Let us know on our social channels. To try xMatters for yourself, race to xMatters Free and use it free forever.

Request a demo