Thursday, August 23, 2007

The Perils of "One Size Fits All" SEM and Log Management Packages


"Smokey my friend, you're entering a world of pain."
-Walter Sobchak, The Big Lebowski



"A Jack of all trades is a master of none"

Today's post is going be a little outside the technical realm of log management, but is an important post nonetheless.

Often, we receive RFPs (requests for proposal) from companies wanting us to run through a "supports/does not support" checklist of log generating devices. It seems that upper management loves to approach enterprise log management as a quest for the one holy grail product that can manage logs from hundreds of different devices and operating systems, in addition to folding the laundry and making coffee.

This approach to procuring log management technology is fatally flawed from the outset.

The thousands of log generating devices and operating systems in today's marketplace truly and completely prevents any vendor from being a polymath at all of them. Some vendors may try to lay claim to supporting tens, or even a hundred of said devices, but often the reality is empty marketing rhetoric without the robust technology present to deliver on the claims.

For example, the level of nuance and detail in the Microsoft Windows ® event log alone is enough to keep a substantially sized development team busy all the time. We can testify to this, as the Microsoft Windows event log is our area of expertise. Multiply this level of nuance and detail by a factor of hundred, or even a thousand, and you have an untenable goal for even the largest of software corporations.

Moreover, value gets diluted very quickly when you start looking at the price tag of "one size fits all" log management packages, especially when compared to picking up a handful of best-of-breed tools that specialize in log management for specific operating systems or devices. Take a hard look at the reporting in one of those mega-SEM packages and see if that "value dilution" is not readily apparent. 10 to 20 log generating devices may be "supported", but reporting will often be limited to a handful of reports per device.

To play devil's advocate for a minute, let's assume that one of these mega-SEM vendors has a very diligent, hard working development team that cranks out new reports as often as possible. What happens when the way an event gets logged on a particular OS changes or a new service pack is applied? Whoops! Back to the drawing board. Patch, patch, patch and fix all of those previously "finished" reports. As the number of reports increases, each new logging change that happens after an OS upgrade or device firmware patch increases that mega-SEM vendor's work by an order of magnitude. Eventually, entropy will take over, making quality suffer while updates are issued in a less timely fashion. It's a battle that cannot be won, even with the best development efforts and the most earnest intentions.

It's tempting for CIOs and CTOs to buy into the mega-SEM hype - the fantasy of having the logs of hundreds of different devices and computers all neatly aggregated with hundreds of ready-to-be-summoned reports at their fingertips. In fact, one can argue that many of these mega-SEM vendors aren't selling software - they're selling the CxO's dreams right back to them. Unfortunately, these dreams are never fully realized. And the results are tragic:


  • Hundreds of thousands, if not millions of dollars, spent on the actual software or appliance

  • More hundreds of thousands spent on service contracts and consulting

  • Lost employee hours attempting to get the behemoth package to work

  • Significant opportunity costs to the business during this process

  • Additional software costs when new vendor packages are purchased to produce the sort of information the mega-SEM package was supposed to be delivering in the first place.

Enough doom and gloom. Here's a novel philosophy that CxOs can use to reduce the pain and maximize the gain of procuring log management technology:

Step 1: Delegate the work of procuring SEM and log management packages to the department heads that manage the different assets of your network (e.g. the Windows Platform team lead, the *Nix Platform team lead, the Infrastructure/Router/Switch/Firewall team lead).

Step 2: Instruct your various department heads to research and test the best-of-breed log management offerings that are directly relevant to the devices and computers they manage. These department heads are in a unique position to understand the subtle details that can sink or swim a particular SEM package in your environment. They can also tell you the role and quantity of the devices they manage, so you can make a more more targeted distribution of resources (e.g. 80% of all managed devices are Windows servers, 15% are *Nix, and 5% are Other).

Step 3: Empower your department heads to procure the log management package that best suits their realm of your network, and make them responsible for managing, operating, and documenting the software, producing reports on a recurring basis that can be directed to you as needed.

It is our contention that if you adopt this approach, your log management project and procured technology will be:

  • Under budget

  • Less prone to failure

  • Less vulnerable to obsolescense or downtime caused by critical changes in event logging

  • More likely to produce higher ROI

Thus we conclude our public service announcement on this topic.

2 comments:

Xavier Ashe said...

Let me introduce myself as someone who implements a "Mega-SEM" as a customer facing managing consultant. As with any software product we have to deal with the difference in the sales pitch and what the customer really wants. Often, the customer has bought the sales pitch to such a degree they have forgotten why they needed the big flexible SEM solution in the first place. It's not necessarily a case of the sale people over selling, it's just a case of getting make to reality when it comes to implementation.

A large software implementation always has the flexibility to do a lot, but takes extra effort to make it perfect for the customer. If you take out the term SEM from you argument, it's one that has been argued for 30 years now. The bottom line is many customers like general solutions that have the flexibility to grow and change as they do.

A proper "mega-SEM" should have that flexibility. The change of logging methods and error codes change constantly, so the solution must be able to keep up with that. Also, I am usually implementing in an environment where they have already followed your device. The Windows team has MOM, the Unix team has Syslog-NG and usually some home grown tools, the Checkpoint team has their NG logging servers, the security team has RealSecure, etc. There's still a role for a cross platform SIM and can show ROI and just a few months.

I'm not a sales guy, I make it work. So once you understand there is a role for the limited best-of-breed products AND flexible enterprise software solutions, I can tell you that implementing a mega-SIM is not doomed for failure.

Well, it isn't if you hire me ;)

Full disclosure: I work for IBM Software Services deploying Tivoli Security Operations Manager and Tivoli Compliance Insight Manager.

Dorian Software Dev Team said...

Xavier,

You make some good points here. And yes, the argument regarding adopting best-of-breed solutions versus the broad enterprise package has been around for a good while.

We still maintain, however, that there are some peculiarities in the log management market that makes the adoption of a "mega-SEM" package very problematic, if not "doomed to failure." For one, the frequent changes to log formats, log schemas, etc make it difficult to stay on top of them if you try and support a whole bunch. Secondly, we don't think the ROI versus TCO on mega-SEM is anywhere near as efficient as what you get from an enterprise antivirus tool for instance.

The myriad of log formats, standards, schemas and the like have prevented the log management market for being commoditized to date, and we think that will remain the case for well into the future. Just look at the scramble now that Microsoft has changed schemas and the whole API to deal with Vista and Server 2008 logs. It's our opinion that mega-SEM vendors are juggling knives trying to stay on top of every change.

We're sure there's room for both in the market (targeted best-of-breed apps versus mega-SEM), but our argument was that you can string together multiple best-of-breed apps and have a greater functionality set with a much lower initial cost and TCO. In this market, we still maintain that is the case.