Loggly helps Rumble Entertainment maximize revenue through a great player experience
Log management is critical for running an operational service like a game. If we didn’t have Loggly, there would be days where our revenue would be impacted by as much as 70 percent.
Albert Ho Executive Producer / Product Manager for Platform, Rumble Entertainment
Highlights:
- Doubled log management capacity and tripled data retention time while reducing monthly costs by more than 75%
- Debugging time cut from days to as little as minutes
- Improved vendor management, planning, and customer service
Challenge
For a game developer like Rumble Entertainment, generating revenue depends directly on two things: its games being up and all of the important features performing well. With three to four releases per week, complex in-game mechanics, and a huge diversity of client hardware and network environments employed by players located throughout the world, this is no small task for Rumble’s team of 40 developers and DevOps professionals. “Our games consume our players’ valuable and limited entertainment time,” says Albert Ho, Executive Producer/Product Manager for Platform at Rumble. “If anything isn’t behaving correctly in our game or if key features are running too slowly, we could lose that customer forever.”
Before Loggly, Rumble built its own log management collector and analysis tool. The project was enormously expensive in terms of developer resources and infrastructure costs, and the service provided only partial visibility into Rumble’s application performance. “Eventually we realized that log management was the poster child of ‘undifferentiated heavy lifting,’” Albert reports. “Getting good at log management wasn’t going to make us better game developers, so we were better off finding a service to take it off our developers’ hands and focusing on our core competency.”
Why Loggly?
Rumble investigated several cloud-based log management services and quickly settled on Loggly. “We realized that we could double our capacity and triple our data retention time while reducing our monthly costs by more than 75 percent,” Albert comments.
Albert cites the the following Loggly advantages:
- The ease of setting up the REST API. “Our developer was able to get game logs into Loggly within one hour of starting our free trial.”
- Loggly’s focus on customer self-sufficiency. “We don’t have time to have someone come in to show us how to use tools or set up dashboards. We just wanted to do it all ourselves, and Loggly’s business model fits perfectly with that vision. We were self-sufficient from the start and running at full power within two weeks.”
- Undifferentiated heavy lifting. “We spent weeks trying to build our own logging and indexing solution for large JSON strings. Our volume of data is so high that it cost us ten times more than Loggly, and our data retention time was three times worse.”
Solution
Using Log4J and Loggly’s simple REST API, Rumble sends its logs without an agent. The company’s developers were able to get the solution up and running within a day. All of Rumble’s logs adhere to a common JSON structure to facilitate the richest analysis and search so that Rumble can quickly troubleshoot the player experience.
Post-Release Quality Management Cuts Costs and Protects Revenue
Rumble sends all stack traces and error exceptions to Loggly. The company uses this data to track a set of key metrics that serve as indicators of game health such as error counts and response times. The company also collects performance data (e.g., lags) for a sample of its players to aid in finding more difficult-to-find problems such as slow performance. Finally, it logs platform problems such as failed transactions or external vendor exceptions.
With every release, Loggly enables Rumble to compare post-release and pre-release metrics; as a result, Rumble can often detect potential problems within minutes. If there are any material changes in the key error metrics, the development team uses Loggly to isolate the cause. Albert remarks, “We can usually push a hot x within one to two hours. More traditional game development practices may rely on day-old revenue figures, use ‘black-box’ testers, or worse, rely on their community to let them know about problems.”
“Before Loggly, we might not find out about problems, period… or until a groundswell of player complaints on Facebook sent us into firefighting mode,” Albert says. “Then, it might take us several days of hunting logs on local servers and writing Unix shell scripts to find the cause. Loggly has cut our debugging time from days to as little as minutes. And now we are the first, not last, to know.”
More importantly, every undetected problem, whether it is a broken feature or slow performance, affects player experience and puts revenue at risk. According to Albert, “If we didn’t have Loggly, there would be days where our revenue would be impacted by as much as 70 percent.”
Responsive Log Management + Proactive Customer Service Drives Player Loyalty
With Loggly data at its disposal, Rumble has taken the next step in creating an excellent player experience: reaching out directly to players who have been affected by certain problems. Once its developers have traced a problem to its source, Rumble can use Loggly to quickly identify the other players that were affected by the same problem. Its customer service team may then o er those players in-game credits to encourage them to come right back to the game.
Trend Analysis Aids in Vendor Management and Planning
Rumble maintains weeks of log data, which allows it to immediately analyze health metrics for the last three to four builds and identify potential areas of process improvement. This data also aids in managing external vendors to their service level agreements (SLAs).
Loggly Is Baked Into the Rumble DNA
The entire Rumble technical staff—development leads, development managers, product manager, testers, and DevOps—now relies on Loggly to keep Rumble’s games running smoothly. In addition, the company has extended Loggly to the third-party developers developing on the Rumble platform. When new developers start working with Rumble, they receive a Loggly account and a JSON message format to use for all of their logs, so that they can share in the benefits of Loggly.
