Vivint Solar applications shine brighter with Loggly

Loggly is a completely managed service. We don’t even have to think about it.

Jeff Brown Software Platform and Operations Engineering, Vivint Solar

Highlights:

• Gains visibility and troubleshooting capabilities for applications running on autoscaling Amazon EC2 environments and serverless AWS Lambda compute services
• Accelerates response times with proactive alerting
• Maintains QoS agreements critical to customer billing with log-based reporting

Challenge: Deliver and Maintain Applications to Drive Business Productivity

Vivint Solar depends on automation and software applications to keep its business running smoothly. These applications support back-office operations as well as outside sales, installation teams, and service personnel for its cost-effective solar energy systems. These teams rely on easy-to-use tools and workflow automation to get their jobs done efficiently.

The 30-person (and growing) software development team at Vivint Solar runs approximately 35 applications in Amazon Web Services (AWS). Most are written in Node.js. The team has used AWS since the company’s inception in 2011, and almost all of these applications send their logs to Loggly.

Why AWS?

Vivint Solar has long taken advantage of the flexibility and agility offered by AWS. Most applications are written in Node.js and run in AWS Elastic Beanstalk today. They use multiple Amazon services including Amazon DynamoDB and Amazon S3. The company has recognized the additional flexibility of the Amazon API Gateway and AWS Lambda and is using these compute services for new applications as well as migrating older applications to them.

Jeff Brown, Director of Software Platform and Operations Engineering, cites the following AWS advantages:

• • • Development speed: “In less than three years, we have grown from four applications to 35. None of these new projects had to wait for servers.”
    • Flexibility: “We can try new things even without any startup delays. You can go from doing nothing to doing whatever you want in about two minutes.”
    • Cost: “There have been no up-front investments needed to get where we are today.”

Why Loggly?

The team at Vivint Solar knew that log management was critical to running secure, autoscaling services. The team chose Loggly because they found it to be easy to use and developer-friendly. “If one thing surprised me about Loggly, it was how easy it was to send our logs there,” says Brown.

Since the Vivint Solar infrastructure includes Elasticsearch, the company considered managing its logs using the Elasticsearch-Logstash-Kibana (ELK) stack. However, a more detailed cost analysis showed that Loggly was the less expensive solution. “It’s not complicated to build a log infrastructure in Amazon EC2,” Brown reports. “But when you combine the EC2 costs and the management overhead, ELK is more expensive than the cost of a Loggly subscription.”

Solution

The majority of Vivint Solar’s Node.js applications, as well as infrastructure based on CouchDB and Elasticsearch, send their logs to Loggly. Loggly also aggregates system logs. The primary users of the system are developers, who can search and filter logs for troubleshooting and debugging purposes. Vivint Solar uses Loggly tags and source groups to differentiate logs from different applications. It has modified and augmented standard Node.js logging libraries to make it easy for its developers to implement tagging.

Responding Faster, Supporting a DevOps Mindset

In addition to reactive troubleshooting, Vivint Solar has created alerts for key error messages and conditions. These alerts go directly to the responsible developers. “Loggly makes it easy for developers to get access to production logs so they can truly take responsibility for their own code,” says Brown.

As Vivint Solar increases its usage of AWS Lambda functions, it has begun managing AWS Lambda logs in Loggly. Although AWS Lambda automatically logs to Amazon CloudWatch, the team has limited troubleshooting capabilities inside CloudWatch. Brown’s team has written a dedicated Lambda function that serves to pull logs from all of its other Lambda functions and send them to Loggly.

“A serverless architecture makes it really inexpensive to run simple functions and to scale them. But you need visibility into what’s happening.”

Managing Business-Critical QoS Agreements

Vivint Solar maintains interfaces to a number of third-party APIs that it uses to run its business. For example, it works with several inverter companies that provide data on consumers’ electrical generation via their own clouds. Since Vivint Solar needs this data for customer billing, it has Quality of Service (QoS) agreements with its providers. It logs the details of hundreds of thousands of daily interactions (e.g., response times, how much data was transferred, etc.) and uses Loggly to report on the average response times from different providers and other relevant metrics that measure provider performance and mitigate potential risks to its billing process.

Log Management Is a Must-Have for Modern Software

Brown summarizes, “When you’re working in an autoscaling environment and must restrict production server access for compliance reasons, you need an interface where developers can go to get access to relevant production logs – even if the server those logs came from is long gone.”