This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.
“Life is really simple, but we insist on making it complicated.” The immortal words of Confucius resonate with anyone who has ever tried to glean useful information from log data.
There are consensus-driven definitions of what exactly log analysis is, but a simplified, accessible explanation might be: to organize log entries into a human-friendly display and make business decisions based on what you learn.
1GB of log data is the equivalent of nearly 700,000 pages of text. Sifting through this manually would take nearly three years, or about the same amount of time it would take to read War and Peace 571 times consecutively (debatable as to which fate is worse). And if you consider that at least a few gigs of log data are produced per day even in small organizations, well… cue the data-parsing apocalypse.
Suffice to say, attempting to make sense of all your logs without some kind of automated assistance will lead to information overload, wasted time and serious staff demoralization.
Investing in a dedicated log analysis platform, be it an in-house build or 3rd-party software, should be seriously considered by the C-level powers.
So the solution appears simple: just find or build an automated analysis platform, right?
Just be careful. Adoption of the wrong platform might just exacerbate the issue. Non-intuitive visualizations and statistical views can lead to hours of screen staring and head scratching.
Log analysis platforms should solve the issue of information overload by breaking down the mass of log entries into digestible, relevant groups, and ignoring non-relevant data. Entries might be grouped according to common characteristics or patterns, e.g. the action performed or users performing them. Keep the following functionalities in mind when building your analysis platform or choosing an out-of-the-box solution:
* Broad collection – Make sure that all the log types you need are supported, including logs that originate from different operating systems, device types, and languages.
* Aggregation – Log entries are structured/written in a wide variety of ways, depending on their point of origin. Aggregation helps clear up this chaos by correlating similar logs together based on their shared patterns or characteristics, e.g. the action performed or users performing them. An aggregation feature should have some level of customization, such as saving logs to user-defined groups for ongoing reference.
* Search – A useful search function goes beyond a mere text box that returns matching log data. It’s important that queries return not only search-specific data results, but also allow users to easily access the context of those results, i.e. what occurred immediately before and after in the log.
* Alerts – A system should alert users to repeated error occurrences, system anomalies, or the absence of typical events. Make sure alert settings can be adjusted to different delivery methods, time intervals and priority levels.
* Automation – In most log analysis systems, the above features are dependent on manual operation and programming by users. But the growing scale and complexity of log data has pushed the most advanced analysis platforms to embrace machine learning. A productive platform should have the ability to:
- Intelligently learn a business’s routine log flows
- Use that intelligence to automatically catch anomalies and errors as they occur in real time
- Generate alerts independently
- Provide a blueprint for devs to resolve identified issues
- Present data trends in intuitive visualizations
Technologies may change dramatically from year to year, but time still equals money. Less time spent manually hunting bugs and system events leads to earlier and more rapid releases, more time for innovative feature development, and happier users. Those are the essential benefits from the DevOps perspective, but log analysis is a diverse art, and has a surprisingly broad range of uses.
Who does it help beyond DevOps and IT?
A flexible log analysis system could conceivably become the singular source of analytics for an organization, across departments. Here are three (among many) examples:
- Security Engineers can exploit an Analysis platform’s search tools to find suspicious actions or breaches. The ideal platform should save search queries for repeated use, and automate the searches for a regular interval of security checks.
- Compliance – Log analysis can help keep your product in line with a range of compliance laws, and a strong automation component can speed up the process of auditing and review.
- Marketing and UX – Tools like Google Analytics and HubSpot might generally be considered irreplaceable for usage and engagement analytics. However, log analysis provides the same essential data: user traffic, referrals, usage time, click rates – they’re all in the log.
Though some departments will be resistant to experimenting with what they perceive as a dev-only tool, the C-suite’s collective ears might perk up at the mention of paying for just one multi-purpose analytics platform. This cuts down operational costs associated with buying a litany of analytics softwares for different teams.
To build or buy?
The market is flush with out-of-the-box log analysis systems, varying in levels of price, quality and versatility. Be mindful of the level of technical support offered by each, particularly if you plan to use the platform for departments without a dev background.
For those larger organizations that have the time, human capital and financial resources necessary to build in-house, this approach allows for made-to-measure customization for your particular industry and your business’s tech architecture. If this is your first time building in-house software, ponder this statistic during your cost-benefit analysis: the industry of data analytics enterprise software will reach nearly $200 billion in sales volume in the next two years. This is indicative of a general (though not exclusive) preference toward commercial software as opposed to in-house.
Not everyone in your organization has prior familiarity with log analysis in and of itself, but they’re already familiar with its essential goal. In the world of big data, log analysis is merely another reflection of a universal business need today -- that is, to take mass amounts of information our technical environments generate and distill it to the essential info you need in order to constantly improve your product.
As we opened with a quote from a scholarly icon of Eastern philosophy, I’ll close with one from a more mainstream emblem of the East: “It is not a daily increase, but a daily decrease. Hack away at the inessentials.” – Bruce Lee
To learn more visit Coralogix.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.