Cut the Noise with Notification Flood Control

There’s a scene in Office Space when Peter Gibbons tells the Bobs, “When I make a mistake, I have eight different people coming by to tell me about it.” It’s a favorite scene among IT geeks because we’ve all worked under the absurdity of bureaucracy run amok.

Something similar can happen in an automated environment, where one event can create dozens or hundreds of individual points of failure that produce separate notifications. With devices buzzing and blaring incessantly, it’s almost impossible to work on issue resolution at an appropriate rate.

Since most of the notifications are routine and often duplicate information, it’s easy for service technicians to tune them out or even route them into a folder for possible review later. This is a dangerous practice because one of those thousand emails could be a new event even more urgent than the last, leading to a major incident if left unchecked.

It’s happened before. In 2013, a major U.S. retailer received so many alerts that its service desk employees missed the alerts about a breach. The breach cost the retailer more than $20 million.

And it could happen to you. Imagine there’s a power outage, and every one of your 129 neatly arranged servers goes down at once. Obviously you want to learn about the outage immediately, but after the first few alerts about individual servers being offline, there’s no value in receiving separate notifications about each server. In fact receiving dozens of SMSs, voice calls or in-app notifications in rapid succession can effectively lock up your phone and prevent you from using it to help solve the problem.

Notification Flood Control
Notification Flood Control would prevent you from being overwhelmed with multiple notifications from an integration when you’ve already been notified and might be working on resolving it.

So, you might ask, how can you get Notification Flood Control? One answer is to be an xMatters customer. Here’s how it works.

If Notification Flood Control is enabled for an integration, and the integration generates two or more notifications for voice, SMS, or mobile app devices in a five-minute period, xMatters delivers the first notification to your preferred devices. xMatters then suppresses any additional notifications for the same device type. xMatters will continue to deliver email notifications and messages to your xMatters inbox, but without melting your phone.

Inbound integration in Notification Flood Control

When enabled, Notification Flood Control applies to notifications that target users directly, via subscriptions, or through activity updates. The Events report and Log Audit report indicate events where notifications were suppressed.

Configurable notification flood control settings
Notification Flood Control settings are configurable, so you can change the default parameters and (for example) choose that on-call staff will receive two notifications before suppression kicks in and then won’t be notified again for another 15 minutes, or whatever works for your organization. You can also choose which device types should be covered by the flood control, and for example include email. All events will still be visible in the xMatters Inbox.

Here’s what you’ll see if you click the flood control menu item on the Developer tab:

Flood Control configuration

The configuration settings for this feature include:

• Notification Limit: the maximum number of notifications to be sent for an integration during the suppression duration.
• Suppression Duration: the time period during which additional notifications are suppressed, after the notification limit has been reached.
• Device Types: the device types that notification flood control applies to (email, the xMatters mobile apps, SMS, or voice).

Remember that for the settings to take effect, you’ll need to enable Notification Flood Control for your integration. We advise enabling Notification Flood Control only for automated systems, e.g. network monitoring, not for service desk systems where incidents are created manually as notifications from these systems are less likely to be strongly correlated.

Request a demo