In Theory And In Reality

I will build a car for the great multitude. It will be large enough for the family, but small enough for the individual to run and care for. It will be constructed of the best materials, by the best men to be hired, after the simplest designs that modern engineering can devise. But it will be low in price that no man making a good salary will be unable to own one-and enjoy with his family the blessing of hours of pleasure in God's great open spaces."

-- Henry Ford

"The greatest improvement in the productive powers of labour, and the greater part of the skill, dexterity and judgement with which it is any where directed, or applied, seem to have been the effects of the division of labour."

-- Adam Smith, The Wealth Of Nations

Not surprisingly, our last post on the perils of "One Size Fits All" log management got a heated response from a blogger whose company tilts at the windmills of "mega-SEM" log management. We were called "profoundly stupid," "naive," "incompetent," and "idiotic." We were happy to receive such high praise for our company, which has been producing software in the log management niche since 1997, over twice as long as many of the johnny-come-latelies into the market. Obviously, we're doing something terribly wrong over here :)

Interestingly, the meat of our post, namely that you can put together a good log management system by combining best-of-breed packages that target different types of logs, was not rebutted. Arguably, it is pretty easy to pull some quotes from a blog posting without actually debating the core philosophy or issue. However, we're not really interested in debating this issue, because it would probably devolve into some sort of academic exercise with plenty of jargon and buzzwords that probably don't mean a hill of beans to you, our gentle readers.

One of the most interesting things in the software industry is the disconnect between the "wouldn't it be awesome if?..." theory and the ugly reality of the marketplace. Nowhere is this more painfully obvious than in the area of SEM and log management. In that spirit, and in the spirit of when academia meets reality, we're going to flesh out our previous blog posting into a little thing we call "In Theory and In Reality."

In theory, every possible device, operating system, or program that generates a log would adhere to a common schema or format when doing so. It seems that every year some new working group releases a paper or proposal detailing that very thing.

In reality, only some devices, operating systems, and programs that generate logs adhere to a common format. Cynically or not, vendors of said devices, operating systems, and programs have discovered that there is money to be made selling consulting services and reporting packages for logs written in their proprietary formats. Some of the most popular logging formats, such as Windows EVT files, syslog, and the W3 logging format have gotten that way due to widespread industry adoption and market penetration, not the other way around. On top of that, even if a log is written in a common format, the devil is in the details of the event!

In theory, every organization looking to automate log management has a budget for that project in excess of $50K, or maybe even $100K. On top of that, they obviously would want a log management package that claims to manage hundreds of devices, even though they only have 5 Windows servers, 100 Windows workstations, a UNIX mail server, and a router/firewall on their network.

In reality, many of the admins we work with daily are lucky if their management has blessed them with $5K to spend on log management, never mind $50K.

In theory, most organizations want a large, macro view of logging activity and trends happening across their network. Highly detailed information and reporting would be nice to have, but the big picture is fine for right now.

In reality, if organizations cannot produce detailed, OS/device-specific levels of information for their auditors, they fail audits.

In theory, IT departments are well-staffed with highly-compensated admins who have plenty of free time to spend on extensive consulting and training for the log management packages they adopt. Really!

In reality, IT departments are often poorly-staffed with admins forced into reactive, as opposed to proactive, positions. They need easy-to-configure software that can produce detailed levels of information quickly and without much fuss.

In theory, only expensive, over-engineered SEM packages can produce any useful level of correlation between different devices and operating systems.

In reality, many device and platform-specific SEM packages for SMBs can output aggregated log data into mineable formats such as database tables, or pass that data over the fence to another logging platform (e.g. syslog concentrator, etc), where data can be routinely grepped and mined as needed for key IP addresses, ports, etc.

In theory, all large Fortune 500 companies and huge government entities would naturally want to adopt a mega-SEM package, because it's the only thing that can even come close to dealing with their diverse, heterogenous logging environment.

In reality, many large Fortune 500 companies purchase specific device and platform-targeted log management packages to get a detailed handle on logging data within a certain department. Often, this is after they've been sold a bill of goods by the mega-SEM vendor and they're facing the crunch time of an audit.

We now conclude this chapter of "In Theory and In Reality." We'll soon take you back to your regularly scheduled programming.

References Available On Request...

Today we're going to tackle one of the biggest misconceptions among admins about the Windows Event Log. In particular, the notion that it is a.) a self-contained file and/or b.) a text file. It is in fact neither one.

If you drop a saved EVT file into a text editor like Notepad, you will see some text strings, but you're also going to see a lot a gibberish. Together, the readable text and the gibberish make up a bunch of linked data structures. Microsoft calls this structure an EVENTLOGRECORD, and a saved EVT file has a ton of them, one for each log entry in the file. An EVENTLOGRECORD contains a whole bunch of information, including the date/time the log entry was written to the file, the security identifier (SID) of the user performing or logging the action, the category of the event, the source, event identifier, etc.

Earlier, it was mentioned that event log files are not self-contained. For example, the log file doesn't contain the name of the user performing the action - instead, it contains the SID number (security identifier). In other words, a log entry generated by NT AUTHORITY\SYSTEM wouldn't have NT AUTHORITY\SYSTEM as a text string in the EVT file - it would contain the SID "S-1-5-18" represented in binary. In some respects this seems tedious, but since usernames get renamed often in a Windows domain, it makes sense. If you decide to review an EVT file again in the Microsoft Event Viewer many months after it was initially saved, you want to see the usernames the SIDs resolve to at that instant, not what they resolved to months ago. So, any application that consumes or reads event log files needs to be able to resolve that information, typically via the domain controller or the member server/workstation where the log file originated from.

Another wrinkle in the Windows Event Log format is how the Description field - which often contains very valuable information - gets put together. EVT files don't contain all the data that you see in the Description field of events you view in the Microsoft Event Viewer. Instead, they contain the key nuggets of data that get combined together with message frameworks from resource files to form a complete description.

For instance, your antivirus software may elect to write Event ID 20 to the Application Event Log. Event ID 20 corresponds to the following message template in the Antivirus software's message files: Virus %1 was detected in file %2

However, the Antivirus software will only send the two parameters needed by the message framework when it logs the event. In this example, MyDoom for the virus name and c:\dropper.exe for the file.

In other words, your EVT file will only contain MyDoom and c:\dropper.exe ... and it's up to the program displaying the event to you to put together the complete description, which in this case is: Virus MyDoom was detected in file c:\dropper.exe.

Given that event log files contain so many references to information that in many cases can only be obtained from the machine where the files were generated, here are some tips that will serve you well:

1.) If you routinely save your event log files, consider saving them in both the EVT/EVTX format and another self-contained format like comma-delimited text or a database table. The EVT file will contain valuable information like the original SIDs, and the text file/database table will contain a snapshot of how that data was translated at a certain instant in time.

2.) If you uninstall software, you may find entries in your event logs where the Description field cannot be correctly formatted. This is because message files often get uninstalled alongside the software. If you reinstall the software, even temporarily, you should be able to see the full event descriptions again.

3.) Transplanting Windows Event Log files to a network outside the one they originated from is a losing proposition. When the Microsoft Event Viewer or other log viewing application attempts to resolve the many references in the file, it will more often than not fail and produce incomplete data. As a result, you will only see limited amounts of information, such as Well-Known SIDs and events from software/hardware that match the software/hardware installed on the originating machine.

Seeing Crimson...

Sorry, couldn't resist that little play on words for this inaugural blog post. We're wrapping up development right now on new libraries to support the management of logs on Microsoft Vista. In case you haven't heard, Microsoft has introduced a new logging format in Vista, EVTX files. Back in Vista's beta days, this logging format was code-named Crimson.

So far, the development experience has been quite an eye opener. While the ReportEvent function, which is the cornerstone function for writing to the event log in the legacy EventLog API, works great in Vista, other Legacy API techniques do not. For instance, if you try calling the OpenBackupEventLog function on Vista to open a saved legacy EVT file, the function will fail. Interestingly enough, Microsoft has still not updated its documentation at MSDN to reflect this problem as of this writing: http://msdn2.microsoft.com/en-us/library/aa363671.aspx

Going in the other direction, legacy Windows clients (e.g. NT/XP/2000/2003) can open a handle to a "live" Crimson/EVTX log on Vista remotely, but the traditional techniques used to parse through and render the data on such a log will most likely fail due to a variety of other reasons that relate to the hardening of Vista's networking and new message provider data stores. On top of that, legacy Windows clients simply have no mechanism for reading saved Vista EVTX log files whatsoever.

While the Crimson/EVTX format does confer advantages over its predecessor, such as XPath queries, we're still not sure why Microsoft elected to cripple the OpenBackupEventLog function on Vista while supporting other legacy EventLog API calls. Certainly, from a programmatic standpoint, it would appear that the function does little more then read linked data structures out of a saved binary file. One perhaps controversial theory is that Microsoft wanted to make transitioning to Vista much easier for publishers of Windows events (e.g. software developers whose programs write to the log), than for the consumers of the those events (e.g. the utility software vendors whose programs manage and analyze log files). Considering Microsoft's efforts to increase market share in the server management market, that could be the case.

Regardless, we've found cool new ways to work around these potential trouble spots and look forward to introducing Vista/Longhorn compatible versions of our software very shortly.