In Theory And In Reality

I will build a car for the great multitude. It will be large enough for the family, but small enough for the individual to run and care for. It will be constructed of the best materials, by the best men to be hired, after the simplest designs that modern engineering can devise. But it will be low in price that no man making a good salary will be unable to own one-and enjoy with his family the blessing of hours of pleasure in God's great open spaces."

-- Henry Ford

"The greatest improvement in the productive powers of labour, and the greater part of the skill, dexterity and judgement with which it is any where directed, or applied, seem to have been the effects of the division of labour."

-- Adam Smith, The Wealth Of Nations

Not surprisingly, our last post on the perils of "One Size Fits All" log management got a heated response from a blogger whose company tilts at the windmills of "mega-SEM" log management. We were called "profoundly stupid," "naive," "incompetent," and "idiotic." We were happy to receive such high praise for our company, which has been producing software in the log management niche since 1997, over twice as long as many of the johnny-come-latelies into the market. Obviously, we're doing something terribly wrong over here :)

Interestingly, the meat of our post, namely that you can put together a good log management system by combining best-of-breed packages that target different types of logs, was not rebutted. Arguably, it is pretty easy to pull some quotes from a blog posting without actually debating the core philosophy or issue. However, we're not really interested in debating this issue, because it would probably devolve into some sort of academic exercise with plenty of jargon and buzzwords that probably don't mean a hill of beans to you, our gentle readers.

One of the most interesting things in the software industry is the disconnect between the "wouldn't it be awesome if?..." theory and the ugly reality of the marketplace. Nowhere is this more painfully obvious than in the area of SEM and log management. In that spirit, and in the spirit of when academia meets reality, we're going to flesh out our previous blog posting into a little thing we call "In Theory and In Reality."

In theory, every possible device, operating system, or program that generates a log would adhere to a common schema or format when doing so. It seems that every year some new working group releases a paper or proposal detailing that very thing.

In reality, only some devices, operating systems, and programs that generate logs adhere to a common format. Cynically or not, vendors of said devices, operating systems, and programs have discovered that there is money to be made selling consulting services and reporting packages for logs written in their proprietary formats. Some of the most popular logging formats, such as Windows EVT files, syslog, and the W3 logging format have gotten that way due to widespread industry adoption and market penetration, not the other way around. On top of that, even if a log is written in a common format, the devil is in the details of the event!

In theory, every organization looking to automate log management has a budget for that project in excess of $50K, or maybe even $100K. On top of that, they obviously would want a log management package that claims to manage hundreds of devices, even though they only have 5 Windows servers, 100 Windows workstations, a UNIX mail server, and a router/firewall on their network.

In reality, many of the admins we work with daily are lucky if their management has blessed them with $5K to spend on log management, never mind $50K.

In theory, most organizations want a large, macro view of logging activity and trends happening across their network. Highly detailed information and reporting would be nice to have, but the big picture is fine for right now.

In reality, if organizations cannot produce detailed, OS/device-specific levels of information for their auditors, they fail audits.

In theory, IT departments are well-staffed with highly-compensated admins who have plenty of free time to spend on extensive consulting and training for the log management packages they adopt. Really!

In reality, IT departments are often poorly-staffed with admins forced into reactive, as opposed to proactive, positions. They need easy-to-configure software that can produce detailed levels of information quickly and without much fuss.

In theory, only expensive, over-engineered SEM packages can produce any useful level of correlation between different devices and operating systems.

In reality, many device and platform-specific SEM packages for SMBs can output aggregated log data into mineable formats such as database tables, or pass that data over the fence to another logging platform (e.g. syslog concentrator, etc), where data can be routinely grepped and mined as needed for key IP addresses, ports, etc.

In theory, all large Fortune 500 companies and huge government entities would naturally want to adopt a mega-SEM package, because it's the only thing that can even come close to dealing with their diverse, heterogenous logging environment.

In reality, many large Fortune 500 companies purchase specific device and platform-targeted log management packages to get a detailed handle on logging data within a certain department. Often, this is after they've been sold a bill of goods by the mega-SEM vendor and they're facing the crunch time of an audit.

We now conclude this chapter of "In Theory and In Reality." We'll soon take you back to your regularly scheduled programming.

The Perils of "One Size Fits All" SEM and Log Management Packages

"Smokey my friend, you're entering a world of pain."
-Walter Sobchak, The Big Lebowski

"A Jack of all trades is a master of none"

Today's post is going be a little outside the technical realm of log management, but is an important post nonetheless.

Often, we receive RFPs (requests for proposal) from companies wanting us to run through a "supports/does not support" checklist of log generating devices. It seems that upper management loves to approach enterprise log management as a quest for the one holy grail product that can manage logs from hundreds of different devices and operating systems, in addition to folding the laundry and making coffee.

This approach to procuring log management technology is fatally flawed from the outset.

The thousands of log generating devices and operating systems in today's marketplace truly and completely prevents any vendor from being a polymath at all of them. Some vendors may try to lay claim to supporting tens, or even a hundred of said devices, but often the reality is empty marketing rhetoric without the robust technology present to deliver on the claims.

For example, the level of nuance and detail in the Microsoft Windows ® event log alone is enough to keep a substantially sized development team busy all the time. We can testify to this, as the Microsoft Windows event log is our area of expertise. Multiply this level of nuance and detail by a factor of hundred, or even a thousand, and you have an untenable goal for even the largest of software corporations.

Moreover, value gets diluted very quickly when you start looking at the price tag of "one size fits all" log management packages, especially when compared to picking up a handful of best-of-breed tools that specialize in log management for specific operating systems or devices. Take a hard look at the reporting in one of those mega-SEM packages and see if that "value dilution" is not readily apparent. 10 to 20 log generating devices may be "supported", but reporting will often be limited to a handful of reports per device.

To play devil's advocate for a minute, let's assume that one of these mega-SEM vendors has a very diligent, hard working development team that cranks out new reports as often as possible. What happens when the way an event gets logged on a particular OS changes or a new service pack is applied? Whoops! Back to the drawing board. Patch, patch, patch and fix all of those previously "finished" reports. As the number of reports increases, each new logging change that happens after an OS upgrade or device firmware patch increases that mega-SEM vendor's work by an order of magnitude. Eventually, entropy will take over, making quality suffer while updates are issued in a less timely fashion. It's a battle that cannot be won, even with the best development efforts and the most earnest intentions.

It's tempting for CIOs and CTOs to buy into the mega-SEM hype - the fantasy of having the logs of hundreds of different devices and computers all neatly aggregated with hundreds of ready-to-be-summoned reports at their fingertips. In fact, one can argue that many of these mega-SEM vendors aren't selling software - they're selling the CxO's dreams right back to them. Unfortunately, these dreams are never fully realized. And the results are tragic:

• Hundreds of thousands, if not millions of dollars, spent on the actual software or appliance


• More hundreds of thousands spent on service contracts and consulting


• Lost employee hours attempting to get the behemoth package to work


• Significant opportunity costs to the business during this process


• Additional software costs when new vendor packages are purchased to produce the sort of information the mega-SEM package was supposed to be delivering in the first place.
  

Enough doom and gloom. Here's a novel philosophy that CxOs can use to reduce the pain and maximize the gain of procuring log management technology:

Step 1: Delegate the work of procuring SEM and log management packages to the department heads that manage the different assets of your network (e.g. the Windows Platform team lead, the *Nix Platform team lead, the Infrastructure/Router/Switch/Firewall team lead).

Step 2: Instruct your various department heads to research and test the best-of-breed log management offerings that are directly relevant to the devices and computers they manage. These department heads are in a unique position to understand the subtle details that can sink or swim a particular SEM package in your environment. They can also tell you the role and quantity of the devices they manage, so you can make a more more targeted distribution of resources (e.g. 80% of all managed devices are Windows servers, 15% are *Nix, and 5% are Other).

Step 3: Empower your department heads to procure the log management package that best suits their realm of your network, and make them responsible for managing, operating, and documenting the software, producing reports on a recurring basis that can be directed to you as needed.

It is our contention that if you adopt this approach, your log management project and procured technology will be:

• Under budget


• Less prone to failure


• Less vulnerable to obsolescense or downtime caused by critical changes in event logging


• More likely to produce higher ROI

Thus we conclude our public service announcement on this topic.