Pulling Back the Curtain on Software Improvement

xMatters engineers work on new product features and software improvement every day. Ultimately, the changes they make are only useful if they work.

Our QA team is responsible for making sure changes to the product are sound – and don’t negatively impact existing functionality. We’re always looking for innovative methods and new tools to improve and test our products.

Successful tools can send intelligent communications when incidents happen. Really successful tools can prevent minor events from becoming major incidents.

One area in which we are always trying to improve is test automation. Many recent xMatters projects have centered on APIs and API-driven software. While our existing automation solutions were sufficient for our various API projects, they were a bit heavyweight and slow to spool up. We simply couldn’t write tests fast enough to meet the demands of our developers.

Enter Runscope
I wanted to explore some new API test automation for one of my 555 projects, which provide us an opportunity to explore non-core projects. Eventually I came across Runscope. Runscope provides a simple UI for creating, capturing and inspecting API requests. The response to any API request can be saved, parsed and utilized in further requests (often using simple tools in the UI), allowing for the creation of tests that exercise an API in an end-to-end fashion.

Let’s look at a sample test. In my test, I wanted to verify that the xMatters communication API returns the correct response (400) when a request targets an invalid phone number. For this test, I posted a request to the API using a known invalid country code and phone number combination, and asserted that the API spits the correct error back.

We simply entered the request URL, along with the preferred method (POST/GET/DELETE etc). We could set any required headers, as well as specifics for the request body.

A tester simply inputs the request URL, along with the method they want to use (POST/GET/DELETE etc).

Any required headers can be set, as can specifics for the request body.

Basic assertions can be done in the interface. In this case, we want to ensure that a 400 is returned, and check that the correct error message is sent back to the caller.

We created basic assertions right in the interface. In this case, we wanted to ensure that a 400 is returned, and check that the correct error message is sent back to the caller.

Using the basic building blocks in Runscope, I automated a suite of tests that performed notifications through xMatters. It was fast, it was easy, and it worked.

Solving new problems: Monitoring
After some time using Runscope to automate QA tests for the xMatters communication API, we decided to leverage it to try and solve another problem: monitoring. Our operations team has excellent monitoring tools in place for making sure things are up and running, but we wanted to build something that actually executes the code and detects problems within an environment that is otherwise considered healthy.

Using Runscope’s scheduling interface, we built out a set of tests that touch each entry and exit point to the API on a set schedule. Every few minutes, a different test executes, and any failures notify us via email. By writing tests that had specific assertions for each of the dependent services within the API deployment, we were able to detect when a component that was fully operational was perhaps doing its job incorrectly.

After some time, this end-to-end monitoring solution proved itself. We were able to detect minor issues that would have turned into major incidents for users of our API if we didn’t intervene. We decided to extend this monitoring to many of our xMatters on-demand users with more demanding environments or tricky integrations. Last month, nearly 1 million requests were sent to xMatters deployments, providing end-to-end monitoring of various xMatters services. The monitoring ensures that not only are the services up and running, but they are performing their duties as expected.

Other Projects
Runscope has functionality that we have been able to leverage for other projects.

• Capture requests and inspect packets, allowing for unique debugging opportunities in complicated integrations
• Design load tests for API clients using captured requests: grab all the traffic for a client consuming one of our APIs (e.g. a mobile app), and use that captured traffic as a data model for performance testing.
• Integrate Runscope with xMatters, so that we could call an engineer on the phone via xMatters, to let them know about a problem with xMatters!

More ideas and projects are always coming down the pipe as we look for new ways to enrich our products and improve how we build them. Our work with Runscope is just one example of how engineers at xMatters are looking for opportunities to achieve the goal of software improvement.

Request a demo