If you are hosting your IT infrastructure in a cloud platform like AWS, you probably know how important it is to have a good log management solution in place. A cloud-based infrastructure is elastic in nature: Resources are created and destroyed on demand as needed. However, you also want to keep track of your assets. You want to know when and how your resources are being provisioned and what changes are being made to them. You want to spot any issues before they affect users, or worse, go unnoticed.

That’s where log management capabilities come into play. A good log management solution not only consolidates logs from a number of sources into one central place, but also enables you to run queries on the data and analyze trends with relative ease.

Loggly makes it pretty simple to import and analyze logs from AWS CloudTrail and AWS Config services, as well as S3 bucket logs and SNS topic messages.

In this post, we will explore some use cases for analyzing AWS Config data in Loggly.

Resource Discovery

When enabled for the first time, the AWS Config service gives you a baseline snapshot of VPC- and EC2-based resources in your network. From then on, any changes made to resources like VPCs, EC2 instances, EBS volumes, Security Groups, Network ACLs, etc. would be captured by the AWS Config service. Full history is available via APIs or in your own S3 bucket.

Bringing AWS Config data into Loggly gives you a real-time, visual view of resources being used in your account. It gives you the ability to drill down on individual configuration items and their parameters.

Configuring Loggly to interface with AWS Config service is really simple. Watch this video to see how this works:

Raw AWS Config data can be difficult to trawl through if you have hundreds of rapidly changing resources. It’s even harder to correlate them with other sources of data, such as CloudTrail or application logs.

Loggly Dynamic Field Explorer^TM is a proprietary Loggly feature that breaks down raw log messages into a navigable, tree-like format. Using Field Explorer, it’s easy to start with a broad category first (e.g., a date range or a type of resource) and then drill down from it. Using this approach, it becomes much easier to find what you are looking for.

Change Management

A second log analysis use case is around change management. Centralizing AWS Config events in Loggly makes it easier to determine when a change was made to one resource and how it may have affected other resources.

Making changes to infrastructure resources is an inevitable part of IT operations. Instance sizes sometimes need to be increased to meet performance demands, new ports have to be opened for application traffic flow, or old instances are shut down at the end of application lifecycle—just to name a few examples.

Following the ITIL principle requires any changes to your infrastructure to be planned, approved, and handed over to your operations team after implementation. Part of this compliance also means you have proper checks in place to ensure your network reflects only approved changes.

When enabled, AWS Config forwards any changes to a supported service to a Simple Notification Service (SNS) topic. With Loggly as the central log manager, the SNS notifications are sent directly to Loggly. It’s much simpler to go back to an event from Loggly’s easy-to-use interface and immediately identify unapproved resources that were created or ones running in a region where they are not supposed to.

If you wish to be alerted when changes are made to an underlying service, Loggly has an alerting feature available. You can set up filters on AWS Config events–just like in any other alerting subsystem–and get notified when a particular event occurs.

Troubleshooting

Finally, the most common use case for Loggly is troubleshooting, and data from AWS Config can play an important role in isolating problems with your application.

Sometimes a legitimate change can have an undesirable effect on the network and its applications. A server may have been shut down as part of a monthly maintenance cycle without you realizing that it was hosting a critical component of the application suite and would take the whole application offline. Retrospective investigation from the date of the incident can reveal the change responsible.

By analyzing AWS Config events in Loggly, you can look for answers to questions like:

• Does an issue coincide with one of my resource-configuration changes?
• Could an unrelated configuration change have contributed to a broad performance issue?

Another example of troubleshooting has to do with security compliance. If there is a breach in application security, retrospective analysis again becomes a necessity. Your network security team may require periodic status reports on access control points like route table entries, security groups, VPNs, Internet gateways, and network ACLs. Changes to these configs can be highlighted very easily when using a solution like Loggly.

If you think there are certain queries you need to run on AWS Config events more often than others, you can save them for reuse.

Conclusion

It only takes a few minutes to try out these use cases. Create a free Loggly account if you haven’t done so already and follow this step-by-step guide on how to send AWS Config logs to Loggly.

About Sadequl

Sadequl Hussain is an IT pro with more than 15 years of experience in application development, database administration, training, and technical writing. He is also an AWS Certified Solution Architect (Associate Level). He loves anything related to database and system administration (both Windows and Linux). At the moment, Sadequl is working with cloud and DevOps technologies and learning NoSQL/Big Data technologies like MongoDB. He enjoys technical writing, presenting, and training.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.