DevOps at Red Robin: Building a Serverless Ordering System

They say restaurants are a tough business. More than half fail in the first year.

So I feel very lucky to be part of Red Robin, one of the most successful casual dining restaurants in the United States.

Founded in Seattle in 1969, Red Robin generated more than $1B across nearly 500 locations last year. That’s a lot of yummy Whiskey River BBQ hamburgers.

Red Robin is an iconic American restaurant, founded by a man who loved barbershop quartets, but tradition hasn’t held our technology organization back.

Being able to incorporate the possibilities of dine-in, and online ordering along with catering offered on one platform, folded into a single architecture is in my opinion, simply amazing. So what does this mean? Simplified architecture that share common services with a given cloud provider significantly improved fault tolerance (Look mama, “No physical disks, disk arrays nor switches!”) as well as scaling for both performance and capacity needs. Moreover, because of this, we are truly able to embrace and implement an agile framework such as Kanban or Scrum which — with all in unison — allow Red Robin to successfully experience a multiple-week software release cycle.  All in all, it is a great example of watching a well-instrumented DevOps practice that you observe and distinctively see the net gain benefit of the business’ bottom line.

Why serverless?

The COVID-19 quarantine really highlights the importance of a serverless ordering system. We launched this infrastructure into production on March 9; a week later, all our dine-in facilities closed, and online ordering became our main pipeline for revenue.

As customers realized the quarantine was more than a temporary inconvenience, we witnessed daily spikes in traffic to the RedRobin.com website. As the demand of online traffic increased, we saw the serverless infrastructure scaled to capacity during peak periods without incident or performance hits, and again scaling back to minimum resource needs as designed. This was all infrastructure as code (IaC) and software as a service (SaaS). There were no emergency change controls to add more hardware for capacity in order to suddenly react to an ever-changing demand of our customer base.

When it came time to modernize our online ordering system, we needed more than a cloud infrastructure. We needed a flexible framework that would help us move fast and effectively. Here are three recommendations for accomplishing great things with few resources:

A diagram of the Red Robin Online Ordering Application.

#1 Have an amazing team

I work with a fantastic team of engineers—I wouldn’t call them developers. I would call them Code Ninjas because every time they hit the keyboard they’re killing it. The infrastructure we’ve built will contribute a significant additional revenue stream for the business using only code as infrastructure.

As a veteran enterprise system management engineer, using application such as HP Omi and Network Node Manager, Solarwinds or similar monitoring platforms, I have worked at some very traditional companies that were heavily invested in very large on-premise server farms that required significant “Heavy lifting,” to execute significant change that would cascade across hardware, networking, and application. However, in comparison to “Serverless Architecture” it is nothing short of astounding when considering its near effortless approach to instantaneous configuration changes through horizontal or vertical scaling.

Looking at everything we’ve accomplished; I am amazed that our new ordering system has absolutely no physical footprint. As a former Enterprise System Management professional who has had the responsibility of monitoring various networked infrastructure, a core commonality among it all was static IP addressing, which meant a monitoring instrumentation tool such as an agent could be installed on a given server or network traffic collection for a given node.  However, with serverless infrastructure the need for a specific node designation is no longer needed as faults detected on the cloud provider’s side trigger an internal process that will terminate and replace a given node that restores only the need underlying resources which is not dependent on specifics such as IP Address. As a result, this removes much of the needed external monitoring infrastructure toolsets which can be substantial cost saving in operational overhead.

The infrastructure we’ve built will contribute a significant additional revenue stream for the business using only code as infrastructure.

#2 Find and nurture champions of agile

Now as we ascend in the era of Serverless Infrastructure (SI), the emphasis is now is less on server infrastructure management and more on platform capability and capacity thanks to containerization. Therefore, our software engineers are solely dedicated to application development utilizing agile methodologies yields short iterations, and regular point releases. Moreover, with the use of cloud services such as AWS Lambda Functions, API Gateways and CloudFormation, along with “Ninja Level Code Assassin’s,” simply adds to the allowance of swift application development of our overall online ordering system.

Moreover, because of the flexibility of SI and the xMatters expansible library of integrations and API endpoints, we were able to create a significant toolchain collaboration between various industry standard application such as Cloudwatch, Slack, Splunk and ServiceNow. This collaboration allowed us to proactively triage events by utilizing xMatters intelligence logic to conditional route events to the expected application end point. This has allowed us to create a very comprehensive feedback loop which provided critical information to mitigate issues and plan for future feature releases. Thanks to the integration with xMatters, we’re working to never miss an order of your Southern Charm Impossible Cheeseburgers.

The biggest factor is to foster true champions of agile, to give them the space to produce in the way that agile produces.

One of the things that slows the transition to agile in traditional businesses is the person that says, “Hey, I need this product. I need it now. Give me an end date.”

But the champion of agile says, “Hey, we’re going to roll out point releases. We’ll figure out the highest priority features, and we’ll release MVPs according to that prioritization.”

So we’ll roll out features per the normal process—in sprints or in a Kanban structure. But we allow people to actually produce.

That’s why it’s essential that management steps back and allows the team to do what they know how to do. Projects like this one prove that small teams leveraging agile can move very quickly and be highly productive, even in traditional businesses like Red Robin.

To drive collaboration and IT communication, we implemented xMatters. And thanks in part to xMatters, our engineers have made our new ordering system a fully manageable application.

#3 Be prepared for change

When I started my engineering career, I had to scour technical manuals and runbooks just to set up a server. A lot has changed in the last 20 years. Today, I can spin up as many servers and containers as I want in a few minutes.

So be prepared for change. Be prepared to reinvent yourself as often as you need to. Invest the time in learning new technology. Build a great team. And there’s no limit to what you can accomplish.

Nothing beats the power of a great team in an environment that drives innovation.

Preston Thornton stands with the sign for Red Robin, where he helped implement a serverless ordering system.

With a solid, agile framework, a small team with great skills and the right mindset can produce high-quality, high-output infrastructure. There are a lot of folks saying, “Hey, we’re devOps, we’re agile.” But they don’t have those processes in place.

So when the business needs to move fast, an agile framework with champions doing the actual work puts the business in a position to succeed.

Red Robin is definitely one of the most collaborative and open-minded environments I’ve worked in. It has made it possible to create an agile framework and operate it within a bigger company.

Order up!