Loggly keeps the Monex Insight development team efficient and focused
Being able to quickly pinpoint the source and impact of issues in emergency scenarios is critical. Field Explorer makes this a breeze.
Jonathan Keith Development Manager, Monex, inc.
Highlights:
-
- Saved more than five person-hours per week by replacing an in-house log management service
- Increased the efficiency and effectiveness of debugging and operational troubleshooting
- Decreased time to problem resolution
Challenge
TradeStation Group, Inc. is an online brokerage company based in Plantation, FL working with a leading Japanese online securities brokerage in Tokyo, Monex, Inc. Both are wholly owned subsidiaries of Monex Group, Inc. Together, they develop web applications that serve a variety of audiences, including active traders in the United States and about 1.5 million consumer investors in Japan. Jonathan Keith is the development manager for Monex Insight, the consumer application. It is built primarily on a Windows development stack with .NET front-end and application services and a MySQL database. It runs on an AWS infrastructure to address the scalability and reliability needs of a brokerage application serving a large user population.
Before deploying Loggly, Keith and his team had built their own log management solution using Graylog2. The in-house solution initially met the team’s needs, but Keith was concerned about the resources that went into maintaining it at the expense of other deliverables. “It was a constant thing that we had to think about,” he explains. “The time updating servers and worrying about disk space was time we thought would be better spent focusing on our core competency.”
Why Loggly?
Keith and his team went to AWS re:Invent 2013 with a goal of finding a better log management solution. After evaluating Loggly and Sumo Logic, Monex decided on Loggly and deployed it in early 2014. Keith cites Loggly’s search interface as the key Loggly advantage Monex saw during the evaluation phase. “With Loggly, we didn’t have to caress the search syntax in order to analyze our logs and get the results we were looking for,” he recalls.
Solution
Monex set up nearly all of its environments—including development, QA, staging, and production—to send its logs to Loggly for safe and easy centralization. The company makes extensive use of the ability to tag each log with specific information about which environment it came from and even which process and/or thread. This logging approach makes it extremely simple to hone in on that data later. While the payload of log messages is in plain text today, Monex is working to transition its application logs to JSON as quickly as possible so that the team can tap into more of the automated parsing and analysis provided by Loggly Dynamic Field Explorer™.
Monex also made it a point to not limit access to Loggly for debugging and operational troubleshooting; instead, it offers access to any user seeking insight. Today, Loggly is being used by almost 50 different people across engineering, operations, and quality assurance as well as product owners and technically inclined business analysts. “We’re pretty open about sharing our log data internally,” Keith says. “Anyone who wants to look at the logs can, and we’ll happily take their feedback.” The simplicity of using Loggly Dynamic Field Explorer means that these users don’t require support from Keith’s team.
Loggly Dynamic Field Explorer Makes It Quick to Understand the Impact of a Problem
As Keith and his team have gained experience performing QA follow-up and investigating operational issues detected via Loggly alerts or reported by customers, they now use Loggly Dynamic Field Explorer in combination with the Loggly search functionality. “Loggly Dynamic Field Explorer has improved usability for our troubleshooting use cases,” Keith remarks. “There’s no question that our team is much more efficient and effective.”
The real-time event counts displayed by Field Explorer help the team understand the magnitude of a problem faster. “We use Field Explorer all the time,” Keith comments. “I can immediately see if a problem is occurring on one or two servers or across the fleet. That’s really critical for us to figure out quickly because it can determine the level of our response. In the most widespread cases, we may even need to create a notification to customers.”
The Monex Insight team uses Loggly in conjunction with several visualization tools such as StatsD so they can dive into the root cause of any changes in behavior or pattern to which they are alerted. “Loggly helps us correlate the fact that something happened with what happened.”
Faster Responses (and Better Customer Experience) with Loggly Alerts
Monex uses alerts to immediately detect known critical scenarios such as:
- MySQL connection errors
- Application errors that are indicative of major problems
- Connectivity issues with back-end data sources
“When we see these things, we know that our users are not having a great experience with our application. We want someone to do something right away, 24/7,” Keith says. “After receiving alerts from Loggly, we’ve been able to identify not only which parts of our infrastructure are experiencing a problem but also whether there are specific hosts generating error logs. This allows us to immediately determine and focus responses to the correct areas of our systems.”
Efficient Debugging of a New Back-End System
Keith’s team is currently working on a new back-end system for Monex Insight. The developers have been sending DEBUG-level logs to Loggly so that they can debug their code faster. “In cases where we have tagged threads, you can just put the thread ID into Loggly and follow all of the messages through,” Keith adds.
Loggly Frees Up Valuable Development Resources
Aside from the team’s gains in efficiency and effectiveness, Loggly has increased the capacity of the Monex development team. “It was huge to drop the weight of managing logging servers and taking care of all of the issues of an in-house log management service,” Keith reports. “More importantly, the staff that were spending time on it are now able to focus on more meaningful work. Before, someone spent at least four to five hours per week on log management, and it could take out an entire day of someone’s week when a problem cropped up. Loggly is incredibly valuable to have.”
