Loggly and Docker deliver powerful monitoring for XAPPmedia
Better log management wasn’t something I initially thought of as a benefit of switching to Docker. But the combination of Docker and Loggly is very powerful.
John Kelvie Chief Technology Officer, XAPPmedia
Highlights:
- Standardizes its diagnostic approach with the combination of Docker and Loggly
- Accelerates resolution of software and non-software issues
- Employs proactive alerting to prevent issues from affecting the business
Log Management: A Microservices Must-Have
Like many cloud-based businesses, XAPPmedia understood the value of centralizing its log data to simplify troubleshooting. As it migrated its whole environment to Docker, Loggly evolved into the heart of how the company monitors its service and diagnoses issues.
Docker Makes Scattershot SSH’ing Obsolete
With the move to Docker, John Kelvie, chief technology officer at XAPPmedia, made the decision to lock down all of the machines in its cluster, which runs on the Amazon Elastic Container Service. “We went from having several machines running certain applications to a cluster running all of our containers,” Kelvie explains. “Manual log access for any particular container is impractical because you don’t know where it’s running in the cluster, and it creates security risks.” While XAPPmedia had used Loggly fairly extensively before the Docker migration, the team’s use increased dramatically in the microservices world.
Why Loggly?
XAPPmedia had selected Loggly in 2014 after a brief experience with Splunk Storm. “At first I was skeptical that log management could scale in the cloud, but Loggly won me over,” Kelvie reports. “I liked the pricing. I liked the front end. I liked the fact that I could pull out my phone at a restaurant and debug an issue right then and there.”
Solution
XAPPmedia initially used Loggly to manage its application logs, which contained diagnostics it had instrumented into its code. Now, the company uses Loggly for all of its logging. Everything that comes out of a container, whether through stdout or stderr, gets sent to Loggly via rsyslog. The trickiest part of the implementation was devising an approach to tagging each log event with the container out of which it came, tags for the service it came from, as well as a unique instance ID. According to Kelvie, “Once we solved that, it worked beautifully! We had just vastly improved our diagnostic capability.”
Log Data Enables Sanity Check on New Microservices Deployments
For the team at XAPPmedia, Loggly is the first destination after every new release. With the top-level view provided in Loggly Dynamic Field Explorer™, Kelvie and his developers can ensure that the new services started and that they are not generating any errors.
“We now have an interface through which we can jump in and see what’s happening with our application and with any instance of it–at any time, wherever we are.”
Proactive Monitoring via Loggly Alerts
XAPPmedia has set up a number of alerts so that the team knows immediately when something is not working correctly. These alerts fall into two general categories:
- Exceptions get routed to developers for debugging.
- Non-software issues get routed to the ad operations team. For example, if the service is receiving multiple requests for an ad that doesn’t exist, the operations team might need to work with a customer or replace creative.
In addition to email notifications, XAPPmedia sends some alerts to Slack channels, which are set up as a Loggly endpoint. It complements Loggly with other metrics and monitoring tools such as Datadog.
Quicker Troubleshooting Keeps Revenue on Track
When troubleshooting an issue, the team has comprehensive error and exception data available. It also can trace transactions in cases where data didn’t get processed correctly, even when no exceptions occurred. Finally, XAPPmedia can sculpt certain payloads to analyze all requests and responses, for example when it releases a new API or onboards a customer. “Everyday tasks that used to take as long as an hour now take minutes or seconds – this is valuable time that we get back. And it helps us say “in flow.” The penalty to switch contexts and look at logs when they are all neatly available in Loggly is much, much lower than when we were doing the SSH/tail two-step,” Kelvie says.
Loggly and Docker Together: Standardized Diagnostics
As XAPPmedia has fully embraced microservices, its infrastructure has achieved a level of uniformity that didn’t exist before, allowing the team to leverage Loggly to a greater extent. “When we put Loggly together with Docker, we standardized and brought best practices to all of our diagnostics in a way that I have never before been able to achieve in my 15 plus years as a CTO,” Kelvie says. “Now, I tell developers not to worry about logging. Everything will get taken care of, tagged to the app and instance, and be easily accessed through Loggly. It’s a huge payoff.”
