Loggly Q&A: Big Data 2.0, Logs, and You

Ben Kepes is a technology evangelist, an investor, a commentator and a business advisor with Diversity Limited. Ben covers the convergence of technology, mobile, ubiquity and agility, all enabled by the Cloud. His commentary has been published on Forbes, ReadWriteWeb, GigaOm, The Guardian and a wide variety of publications.

Loggly: You wrote about Big Data 2.0 recently…what is that?

Ben Kepes: In the “old days” of big data, you had the data scientist using complex software to crunch the numbers and then on the other side you had the business analysts using another piece of software for creating charts and making the data pretty. Now, vendors are coming up with single platforms that can help those two groups of people work together. The tools need to be robust enough to manage lots of data but also, the visualization must be really sharp. These new collaboration platforms are going to help by decreasing the time and steps needed to get from data to business value, yet the cultural and human barriers are still more important.

Loggly: What are those barriers exactly?

BK:  You have two different perspectives of the world. The analysts are concerned about making hypotheses, yet the business people want to see what is the impact of big data analyses on customers and the business. Another struggle is that companies don’t always know which questions to ask. Predictive analytics for sales is a hot market, and those applications help companies ask questions like who’s going to buy my product next, which customers will be most profitable, or which products will be in most demand.

Loggly: Has Big Data been too focused on sales and marketing projects instead of looking at other areas of the business such as operations?

BK: Not really. The cloud has also been focused on sales and marketing programs all along. In any space, whether that’s in consumer electronics or cars or clothing, it’s not so much about building a better product anymore but differentiating on your brand. There are so many challenges in markets today with rapid commoditization, and so many homogeneous products. Marketing can help a company rise above the chatter.

Loggly: Do logs play a role here?

BK: Logs are still being used primarily for IT operations and improving IT performance. Yet a bigger opportunity is when we take application logs and correlate them to business performance, such as revenues and profitability. If we’re able to use logs to see the impact to the bottom line of interactions with applications, that’s where it gets really interesting. Google Analytics is one example of this type of application. A company can see traffic patterns, and where people are coming from, and compare with shopping results to make correlations between buyer behavior, customer profiles and sales. This requires integrating and meshing up logs from many sources. I’d like to see companies combine external data such as from social media with backend systems, which could help them the impact on sales from a marketing campaign, or the ability to track the lifecycle of a relationship.

Loggly: Is log management still too hard for most companies?

BK: Yes, it’s still really hard to tie all these data points together and get something meaningful from it. We used to have limited data coming in, so it was easy to understand the patterns. But now, there are huge amounts of data flowing into the systems and the data is not normalized so you can’t make simple correlations. There is no solution on the market today that tells you exactly where you should focus your time and attention. I would love to see a platform where a company could plug all data from any source into it, and it would spit out a metadata analysis showing the top trends. I love the Marc Andreessen message that software is eating the world, but for that to happen it has to be easier to use. If you are a mom and pop hardware store in the Midwest, and you are trying to put together a cloud point-of-sale system and a loyalty application, it’s pretty complex. There’s still too much of the Silicon Valley type of arrogance in the world of software. Technology should be enabling equity for companies, but it’s really not, yet.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.