Log management costs aren’t just data volume and retention time

With the rise in automation, API-based integrations, and microservices, you’re probably dealing with greater log volume, variety, and velocity than ever before. Fortunately, there are also a number of log management options—cloud-based and on-premise—to help you make sense of your logs. Every solution offers unique advantages, but price plays an important role in selecting a log management system.

During one recent interview, a Loggly customer told me, “If the cost of managing logs for a server is greater than 10% of the cost of the server itself, it is a no-go for me.” The question is, what are the components of cost that you need to consider?

SaaS pricing looks simple but…

As a buyer, when you see the pricing page of any solution including Loggly’s, you take it that the log management costs are primarily based on how much log data volume you generate on a daily or monthly basis and how long you need those logs to be retained. Based on our experience serving thousands of customers of different sizes and business models, we suggest you look beyond log data volume and retention when calculating your log management costs.

Vendors may offload six key cost drivers onto you!

The reality is that not all vendors deliver the same types of product offerings. If you need a production-grade log management solution, you need to factor the following six drivers into your total cost of ownership (TCO) comparisons because you may find these tasks offloaded onto you!

1. The cost of unauthorized access: You may need to implement security and access controls to restrict access to logs that contain confidential information. For example, the open-source ELK stack does not offer out-of-the-box security and access control, so it requires you to do some custom work or use Shield, the security add-on from Elastic.
2. The cost of downtime: You may need to develop your own strategy for distributing pipeline and hosting resources in multiple locations to ensure high availability for your log management system. After all, if your log management system only resides on the same infrastructure as your application, it may be inaccessible when you need your logs the most. This is even more important if you are using a lot of containers, since they may churn as much as 9x faster than VMs.
3. The cost of scalability: As your application grows, you may find yourself provisioning servers, adding  clusters, and performing other tasks needed to manage capacity. Adding Kafka instances to manage volume bursts includes additional work and increases log management costs for you.
4. The cost of poor performance: Poor performance affects everyone who uses your log management system including engineering, ops, support reps, and product managers. And it has the potential for bottom-line impact from slower issue resolution and lost revenue. Keep in mind that you may need skilled resources on call to perform highly specialized tasks such as index management and shard allocation.
5. The cost of missing integrations: Likewise, you may bear the cost of skilled resources needed to build and maintain custom alerting or DevOps tool integrations that speed up your troubleshooting tasks.
6. The cost of testing: If you must recreate production like scale and scenarios to fully understand what’s missing, then that testing effort is going to cost you, too.

All ELK providers are not created equal

Remember that even for some cloud-based ELK solutions, you may end up doing a large number of management tasks just like you would do for an on-premise solution.

There’s something similar in ELK, but it didn’t seem to update quickly. It would show fields from a log file you were no longer looking at. We might have been able to fix it, but we just didn’t have the time to mess around. If there’s a solution that already works, as we’ve found with Loggly, and the cost is comparable, then it makes more sense to go with something where we’re not responsible for patching software or rolling out new features.

Jeremy Koerber Senior Systems Engineer, Creative Market

So how do you understand the real log management costs?

When you try out a log management solution, it is very important for you to replicate your daily workflows as much as possible. This means:

• Sending a variety of log data in production volume (not just a single file of old system logs)
• Engaging users from different teams including customer support who will ultimately use the system
• Solving real-world problems, from troubleshooting, daily monitoring, and anomaly detection to longer-term trend analysis

Only then can you see what’s missing and what you need to do yourself.

It’s not complicated to build a log infrastructure [using ELK] in Amazon EC2. But when you combine the EC2 costs and the management overhead, ELK is more expensive than the cost of a Loggly subscription.

Jeff Brown Software Platform and Operations Engineering, Vivint Solar

Now that you have the full picture, next time you visit the pricing page of any vendor, make sure to think beyond volume and retention. Loggly makes Log Management easy for you with simple pricing as well as a full-featured 14-day trial.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.