Why Can't A Windows Server 2008 or Vista Log Be Viewed On My XP Machine?

The following was excerpted from our recent Event Alarm product update announcement:

It seems simple enough, doesn't it? At Dorian, we're seeing the question more and more, and we wish we had a better answer. But - regardless of what log management package you choose - if you want to review an EVTX log (that is, a log generated by Windows ® Server 2008 or Windows Vista ™) you're going to have to open it on a Windows Server 2008 or Windows Vista machine.

Why? Because the new Windows Event Log API functions are only available inside Windows Vista and later operating systems, legacy Windows operating systems like XP and 2003 cannot read previously saved EVTX files at all. There is simply no forward compatibility for consuming saved EVTX files. Period.

And while the legacy Event Log API can be used to read some of the events from an "active" EVTX file (that is to say an EVTX file currently being maintained by the EventLog service on a Vista machine), it cannot properly read and parse some events recorded by the new API.

Many remember when vending machines started accepting paper money. Whenever one actually had paper money, it seemed the "legacy" coin-only machines were all that were around. Try as you might, that XP machine isn't going to read that EVTX log. Don't thank us - thank Microsoft.

Our LogRefiner technology helps manage both formats (EVT and EVTX) side-by-side. Even with this snazzy new technology though, if there are any EVTX logs in the mix, plan on installing our software and managing from a Windows Vista or Windows Server 2008 machine.

Meanwhile, got change for a dollar?

Event Alarm Monitors EVT and EVTX Logs, Side-By-Side!

Event Alarm Version 6 was released in late August. This week, we sent out our official version update announcement.

Just like our prior Event Analyst and Event Archiver releases, this version of Event Alarm is completely Microsoft Vista™ and Windows Server ® 2008 compatible, and features our revolutionary LogRefiner™ technology. You can download Version 6 of Event Alarm here.

We've already mentioned in a bunch of posts that trying to read legacy EVT files on Windows Vista and Server 2008 is quite a chore, with missing fields and information being quite common.

Well here's the good news. Thanks to our pioneering LogRefiner™ technology, you can remotely monitor EVT and EVTX files natively and side-by-side when Event Alarm is installed to a Microsoft Vista or Server 2008 computer. No weird conversions or intermediate steps are necessary, and you get all the data parsed correctly from both log formats the first time. For those admins who are attempting to run Windows Vista or Server 2008 on their workstations, this is a big plus because now you can use Event Alarm as your preferred monitoring solution for all of your Microsoft Windows computers, regardless of how many have been migrated forward to Vista/Server 2008 and the new EVTX format.

On top of Event Alarm's remote, agentless log monitoring, when Event Alarm is purchased as part of Dorian Software's Total Event Log Management Solution™, you effectively have a comprehensive platform for archiving, analyzing, and monitoring event log data from EVT and EVTX log files throughout your network, all from a single install point, network topology permitting.

Here's the full launch announcement for Event Alarm Version 6, complete with a comprehensive feature listing.

Why Your HR Department Will Love Windows Vista, Even If Your IT Department Doesn't.

We're back, gentle readers, with a delicious posting about two new Event IDs available in the Microsoft Windows Vista™ Security Log: Event ID 4802 and Event ID 4803.

Event ID 4802 tracks whenever the screensaver is invoked after a group policy-determined idle time.

Event ID 4803 tracks whenever the screensaver is dismissed by the logged-on user.

Using our versatile Event Analyst® reporting utility, it's easy to create a custom report to track the productivity of your staff.

Here's an example of said report, grouped by user and then sorted chronologically.



In this example, MarkW's screensaver kicked in at 3:04:10 PM and then was dismissed at 3:30:00 PM. Later, the screensaver came back on at 3:45 PM. If your company mandates a given idle time before the screensaver is launched on all desktops via Group Policy, it's easy to calculate the total idle time by adding that number to the period in between Event ID 4802 and Event ID 4803.

For maximum reporting capabilities, consider using our Event Archiver® log collection tool to bring your Microsoft Vista workstation security log data into a central database on a routine basis. Then, link Event Analyst up to said database table, build said custom report, and impress your HR department! Both of these tools are Microsoft Vista and Windows Server® 2008 ready, so have at it.

Finally, we do have a current promotion on Event Archiver, Event Analyst, and Fortress Desktop™ workstation licenses when purchased together. For more details, review our Promotions page for more details.

FYI - For those organizations not running Windows Vista yet, you can still obtain information about screen saver run times by using our Fortress Desktop utility, and then create a similar report in Event Analyst.

A Big Thank You to Our Clients and Partners

While this isn't "development-related" per se, it is a product of our development efforts, so we wanted to share it with our readership.

Dorian Software Posts Highest Quarterly Sales Revenue Ever

A big thanks again to all our clients and partners for their ongoing support.

Mega SIEM/SEM = Mega Headaches

It's been almost a year since we wrote about the perils of mega-SIEM/SEM packages, and now it looks like industry analysts are starting to agree with us.

Take a look at the following Network World article entitled "SIEM tools come up short."

A key quote: "User interfaces were clunky, reports were incomplete, data parsing problems are still around, and when it came to trying to figure out what the heck was going on in our Windows environment, most products left us scratching our heads. (One could argue, however, that this is as much Microsoft's fault as
anyone else's.)"

Ouch! That left a mark.

It's a good thing for those organizations that there is at least one vendor that does Windows log management correctly. :)

We wonder if these mega-SIEM vendors have even gotten a handle on Vista, Server 2008, and the new EVTX log format. Something tells us the answer to that question is "no."

Event Analyst ® 7 Can Slice and Dice Your Security Event Logs ... Any Way Your Auditors Want Them Served

After more sweat and tears, the Dorian Software Development Team is happy to announce the release of Event Analyst Version 7. Version 7 comes with a huge overhaul to the custom reporting engine inside Event Analyst, allowing our users to parse the smallest details out of Windows Security Log Events, grouping and sorting them to their (and their auditors!) hearts' content. We call this Event Analyst's PrecisionParser ™ capability, and it's a subcomponent of our greater LogRefiner ™ technology.

Some log management software in the marketplace attempts to tokenize and normalize security log data at the time of collection/import, necessitating 1.) a database platform for analysis, 2.) numerous table schemas to store the different types of tokens for different categories of events (e.g. taxonomies), 3.) revisions of said schemas as event tokens expand over time (often as a result of new operating systems and service packs).

The whole process above is pretty labor intensive, and if you're a forensic auditor or the administrator of a small network, setting up a database for this purpose can be a costly endeavor. You may just want to open an EVT/EVTX file and rip it down every which way to produce some nifty reports. Or import a handful of said files into Access ®, and then rip them down together.

We have opted for a different approach. Our PrecisionParser does the parsing of key Windows Security Log Description subfield data at the time the data is analyzed and reported against. It can work against a bunch of different formats, such as security log data still inside EVT/EVTX files, to comma-delimited text files and database tables produced by Event Archiver, our log collection and centralization software package.

Yes, you heard that EVTX part right. While some vendors still have their heads in the sand regarding EVTX compatibility for Windows Vista ™ and Windows Server ® 2008, Event Analyst can already parse the EVTX logs just as easily as the EVT versions, even if security log data from both operating systems resides together in one database table. This is a good thing, because the number of security events (as well as the tokens in their Description fields) have only expanded within Vista and 2008.

Here are some of the details on PrecisionParser inside Event Analyst:

As any veteran of security event log analysis can tell you, the subvalue name/data pairs in the Description field of Windows Security events are the golden nuggets that must be mined to generate meaningful reports. Existing users of Event Analyst have already enjoyed the capabilities of Event Analyst's prebuilt reports to extract, group, and sort this level of detail in a variety of categories, like logon activity and group management.

Now, Dorian Software has incorporated its exclusive PrecisionParser capability - a component of Dorian Software's exclusive LogRefiner technology - into Event Analyst's custom reporting engine. What does this mean to you? Plenty! Virtually any type of security event can now have its key subfields parsed out, grouped, and sorted inside Event Analyst's custom reporting engine. Want to group your 529 logon failures by Source IP Address and Authentication Package? No problem. Need to sort file access events by Handle ID? We've got that covered as well.

The benefits of Dorian's PrecisionParser capability are tremendous, and include:

True Log Format Independence - Parsable security log data formats include native EVT and EVTX files, comma-delimited text files produced by Event Archiver and Event Analyst, and Microsoft Access, SQL, or Oracle database tables produced by Event Archiver and Event Analyst. Dorian's multiple log format support stands in stark contrast to other vendor packages, which depend on multiple database table schemas in attempt to normalize log data at time of collection, rather than normalizing data at time of analysis.

True Operating System and Service Pack Level Independence - PrecisionParser can handle virtually all security log data collected from different Microsoft operating systems - from Windows NT 4.0 to Windows Server 2008. This is important as Microsoft frequently expands reported data in security log events over time, often after service packs are applied. If a custom-defined subfield is not present in a legacy operating system event, the custom reporting engine degrades gracefully, simply indicating that the field was not found.

Correlation Across Related, Yet Different Security Events - Correlation is possible among different security events that share common subfields in their descriptions. For example, many security events log handle identifiers, logon identifiers, and IP addresses. Custom reports paired with advanced filters can now be designed to show a variety of event activity that is in fact related via these fields.

Support For Multiple Occurrences of the Same Subfield - While less common in legacy security events, Windows Vista and Windows Server 2008 now often include the same subfield name twice in the Description field. For instance, Event ID 4724 describes the resetting of user passwords by an administrator. Yet the order of the occurrence of the user in the Description determines whose password was reset, and who actually reset the password. When defining custom fields for reports, Event Analyst allows you to make this subtle distinction by indicating if you would like to parse out the second, third, or nth occurence of that field.

Multiple Report Formats Remain Available For Presentation and Data Mining - As in previous versions, custom reports in Event Analyst will continue to be generated in both HTML and CSV formats. The printer-friendly HTML version of the report is excellent for presentation and review by management, whereas the CSV version of the report allows you to import raw, parsed subfield data from the description field into other software packages, such as Microsoft Excel ®. Frequent users of Microsoft Excel will be amazed at the level of analysis possible when reviewing CSV files with Excel's AutoFilter feature.

Importer™ Tool for Event Archiver® Released

One of the more challenging things about log management is trying to collect the vast amount of data that is generated in multi-site networks over limited bandwidth links.

To that end, we have developed a companion tool to our Event Archiver® software - the Importer™ tool for Event Archiver.

Basically, you can instruct all of your various Event Archiver installations to send compressed sets of log data in EVT/EVTX and comma-delimited formats to a computer running the Importer utility. You can use Microsoft Windows file shares or FTP to transport the compressed log file pairs as needed.

Once received, the Importer utility can decompress the log data and automatically import it into a central Microsoft SQL or Oracle database for analysis by our Event Analyst® software.

If you want to deploy a log management solution, but are struggling with the concept of consolidating your data over limited bandwidth pipes, this tool is the answer. It's also a better system than having to deploy an agent to every computer on your network; using the Importer system, you typically only need to deploy one instance of Event Archiver to each local network / branch office.

As far as bandwidth considerations go, by transmitting the data in compressed form, the bandwidth necessary is only 7 to 10% that of the uncompressed log files. We have clients who have successfully used this solution over satellite links, so it has been proven in the field.

For more information on the Importer utility for Event Archiver, including licensing costs, please visit http://www.doriansoft.com/importer

UltraAdmin Version 6 Now Available

As promised, version 6 of UltraAdmin® is now available for download from the Dorian Software website. This version is being made available at no charge to any network administrator who wishes to use it. For those organizations and individuals that need priority support for the product, you can purchase it here.

We've also made available a PDF file listing all of UltraAdmin's features.

The new features in Version 6 include the utility's support for Microsoft Vista™ (important for those admins who are using it on their workstations), support for EVTX and EVT event log reading when installed on Vista, and a new database query tool that can be used to comb through data exported by UltraAdmin into a Microsoft Access database.

A rather nice, but often overlooked feature we introduced in Version 5, is UltraAdmin's ability to manage startup programs on servers and workstations remotely. Specifically, UltraAdmin can manage Run key programs, BHOs, Winlogon notification packages, and Startup folder links. Consequently, UltraAdmin can be used to remove or limit some types of spyware or malware that hook these areas to ensure their load at startup.



For example, an administrator using UltraAdmin can:

1.) Locate the offending executable or DLL that is loaded at startup.
2.) Change the NTFS permissions on that executable or DLL remotely so that no one has access to the file.
3.) Reboot the workstation or server in question remotely.
4.) Delete the offending files after the system has been restarted, and delete the startup hooks referencing those files.

Obviously, some forms of spyware/malware are more tenacious than others, and will be substantially harder to remove. For powerful anti-spyware software, we recommend CounterSpy™ from our friends at Sunbelt Software. Still, UltraAdmin remains an excellent tool in the admin's arsenal, especially for spyware/malware that be surgically extracted in the manner mentioned above.

UltraAdmin Is Now Free, and Features LogRefiner Technology

When you take a bit of a hiatus from blogging, you need to resurface with a bang. To that end, we're excited to announce that UltraAdmin Version 6 will be released next week. The best part?

We're making it available FREE to any network administrator who wants to use it. Fully functional, uncrippled, comprehensive Windows domain and Active Directory management at your fingertips.

We will sell priority support plans for it at $99.00 USD per admin per year, for those who desire that level of assistance.

Some of the biggest Version 6 highlights:

1.) It now supports Microsoft Vista™

2.) It can read both live and saved EVT and EVTX log files when run on Microsoft Vista. Admins won't need to crank up two different versions of the Microsoft Event Viewer to view their logs. We've placed some of our amazing LogRefiner™ technology inside UltraAdmin to accomplish this.

3.) It has a built-in query tool that complements the UltraAdmin Reporter/Exporter module, allowing the administrator to quickly comb through the Microsoft Access databases that UltraAdmin can populate with domain objects and their properties.

We'll post more information on UltraAdmin - including a link to download it and a comprehensive feature listing - early next week.

Tracking Software Installation and Removal Using Event IDs 11707, 11724, and 592

In these days of malware, spyware, and compliance regulations, a lot of admins are looking to track the installation of unauthorized programs, and/or the removal of required programs from client desktops. There are actually several events you can look for in both the Application Event Log and Security Event Log that will help you do this.

In the Application log, setup packages that use the Windows Installer to install themselves will create numerous events, all with an event source of MsiInstaller.

Event ID 11707 tells you when a install completes successfully, and also the user who executed the install package.

Event Type: Information
Event Source: MsiInstaller
Event Category: None
Event ID: 11707
Date: 11/9/2006
Time: 3:21:45 PM
User: DOMAIN\USER
Computer: COMPUTERNAME
Description:
Product: Event Archiver Enterprise -- Installation operation completed successfully.

Event ID 11724 tells you when a software package is removed successfully, again logging the user behind the operation.

Event Type: Information
Event Source: MsiInstaller
Event Category: None
Event ID: 11724
Date: 11/12/2007
Time: 7:50:13 PM
User: DOMAIN\USER
Computer: COMPUTERNAME
Description:
Product: Event Archiver Enterprise -- Removal completed successfully.

You can track both of these events in our Event Analyst software by setting up appropriate filters and building a custom report.

Also, if you want to correlate the name of the executable setup package that was executed to install a piece of software, turn on Process Tracking auditing on the relevant Group Policy Object for one or more computers (e.g. Domain Security Policy, Local Security Policy), and look for events with Event ID 592 in the Security log that occur around the time of the 11707 event in the Application log, e.g.

Event Type: Success Audit
Event Source: Security
Event Category: Detailed Tracking
Event ID: 592
Date: 11/9/2006
Time: 3:20:30 PM
User: DOMAIN\USER
Computer: COMPUTERNAME
Description:
A new process has been created:
New Process ID: 2816
Image File Name: \EvntArch.exe
Creator Process ID: 516
User Name: USER
Domain: DOMAIN
Logon ID: (0x0,0x3E7)

Event Analyst also has a built-in Process Usage report that is very useful for viewing all of the executable files that were loaded and unloaded on one or more systems for a given time frame. It automatically determines the executable files that are run the most frequently for any given user.

Free Software Offer For Early Vista/EVTX Log Format Adopters

As promised in our previous post on Event Analyst's full support for working with EVT and EVTX log files natively on Windows Vista™, we are making a special free software offer available to admins that wish to use Vista as their OS platform for log management.

Here are the details of the offer, directly from our sales division:

Do you already have some Windows Vista machines generating EVTX logs? Great. We'd like to give you some software. That's right. At no charge. We're offering 5 server license packs of Event Archiver™ and Event Analyst™ bundled together. Basic email-based support is included with all licenses. If you wish to pick up an upgrade service or another of our more advanced support options, we can arrange for the purchase. Interested? Simply request more details at
http://www.doriansoft.com/evtxsoftwareoffer.

As you can gather, this is a fantastic promotion, as it allows you to gather event log data from both your non-Vista and Vista systems and report on that data by running Event Archiver and Event Analyst on a Microsoft Vista workstation. We're convinced that once you see the power of Dorian's LogRefiner™ technology in action, you'll be much more comfortable in putting forth a plan for log management for your larger migration to Microsoft Windows Vista and Windows Server 2008™. As we've stated numerous times before, our exclusive LogRefiner technology is here and ready for you whenever that migration begins.

Event Analyst Works With EVT and EVTX Files, Side-By-Side!

Greetings, gentle readers. It's been a while since our last blog post, but that's because we've been slaving on the Version 6 release of Event Analyst. And what a fantastic release it is!

Just like our Event Archiver release of a few months ago, this version of Event Analyst is completely Microsoft Vista™ compatible, and features our revolutionary LogRefiner™ technology. You can download it here: http://www.doriansoft.com/download.

We've already mentioned in a bunch of posts that trying to read saved, legacy EVT files on Windows Vista is quite a chore, with missing fields and information being quite common. In fact, a recent blog posting from the Performance Team at Microsoft shows you how to perform a whole bunch of contortions in an attempt to convert an EVT file to an EVTX file, with of course there being no guarantee that the converted log will parse properly when you attempt to read it.

Well here's the good news. Thanks to our pioneering LogRefiner™ technology, you can work with EVT and EVTX files natively and side-by-side when Event Analyst is installed to a Microsoft Vista computer. No weird conversions or intermediate steps are necessary, and you get all the data parsed correctly from both log formats the first time. For those admins who are attempting to run Windows Vista on their workstations, this is a big plus because now you can use Event Analyst as your preferred log reader/analysis tool/reporting tool for all of your systems and your saved EVT log files. You no longer need to convert EVT files or juggle both the Microsoft Classic Event Viewer and the new Vista Event Viewer when switching back and forth between EVT and EVTX files.

Here's a screenshot of both an EVT and EVTX log being viewed within Event Analyst 6 at the same time:




Again, bear in mind that this technology lets you work with active AND saved EVT files from your older operating systems all natively inside Vista. It's very cool stuff.

We'll have more information for you on this technology soon, including a very nice licensing promotion, so please stay tuned.

New EVTX Log Format Whitepaper Released

Earlier this week, we released a new whitepaper that outlines many of the key changes in the new EVTX event log format found in both Microsoft Vista and Windows Server 2008. We've included some choice pull quotes below, but you can download the full version by registering here:

http://www.doriansoft.com/evtx

Beginning with Microsoft® Windows Vista™ and Windows Server® 2008, Microsoft has completely redesigned its event log format. This new EVTX file format stores event log records as a stream of binary XML records. Accessing data in the new EVTX files requires the use of a new application programming interface that is not available in older Windows operating systems. In addition, the number of, structure of, and data within the fields in the EVTX log records has changed significantly.

Because the new Windows Event Log API functions are only available inside Windows Vista and later operating systems, legacy Windows operating systems like XP and 2003 cannot read previously saved EVTX files at all - there is simply no forward compatibility for consuming saved EVTX files. And while the legacy Event Log API can be used to read some of the events from an "active" EVTX file (that is to say an EVTX file currently being maintained by the EventLog service on a Vista machine), it cannot properly read and parse some events recorded by the new API.

In summary, both forward compatibility to EVTX files from legacy Windows operating systems and backward compatibility to EVT files are severely hampered, if available at all. As a result, organizations that rely on their own scripts and automation techniques may be tempted to develop two different systems for log management - one supporting legacy EVT files on legacy operating systems, and another supporting EVTX files on Windows Vista and Windows Server 2008. Such a strategy has the potential to decentralize log collection and reporting, as well as substantially increase costs over time.

Again, to read the full version, please register here:

http://www.doriansoft.com/evtx

The AUXSOURCE Switch

In a previous post, we discussed how EVT files contain references to other message files, which are parsed together with the EVT data to produce readable events. Typically, when you relocate EVT files away from the network where they were generated, you cannot completely parse all of the data fields properly. However, there is a very useful yet undocumented/underdocumented command-line switch you can use with the Microsoft Event Viewer to force it to look elsewhere for critical message files. That command line switch is /AUXSOURCE

For example, if you had a security log that originated from a Windows® 2003 server, but you were not currently connected to the network where that log came from, you could use the /AUXSOURCE switch to load message data from a Windows 2003 server that was on your local network instead. The command-line syntax would look like this:

mmc /a c:\windows\system32\eventvwr.msc /auxsource=REFERENCECOMPUTER

where REFERENCECOMPUTER is the network name or IP address of the computer that will act as the lookup computer for message file resolution.

Once you load the Event Viewer with the AUXSOURCE flag, you can then open up your saved EVT file, and the Event Viewer will always use the REFERENCECOMPUTER for message file data when it attempts to parse events from the saved log.

There are some caveats with this approach that are listed below:

1.) The AUXSOURCE switch is only available for use on Windows XP and Windows 2003 versions of the Event Viewer, not Windows 2000 versions.
2.) AUXSOURCE will not help you properly view saved DNS Server, Directory Service, or File Replication Service logs from a Windows XP workstation or Windows 2003 member server, even if you point the REFERENCECOMPUTER to a domain controller. Instead, you have to be logged on to a Domain Controller to view these saved files.
3.) If you use AUXSOURCE with Application or System logs, you may still get incomplete Description fields, because chances are the REFERENCECOMPUTER will not have all the same software and hardware installed as the machine where the EVT file came from.

Fortunately, we have decided to provide functionality that exceeds what the /AUXSOURCE switch can do in the upcoming release of Event Analyst. The new version of Event Analyst will allow you to use any Windows machine available on the network (e.g. Windows NT, Windows 2000, Windows XP, Windows 2003) as a reference computer for message files for saved EVT files. No minimum OS platform is required for this functionality - Event Analyst can be installed on Windows NT 4.0, Windows 2000, Windows XP, Windows 2003, etc.

In Theory And In Reality

I will build a car for the great multitude. It will be large enough for the family, but small enough for the individual to run and care for. It will be constructed of the best materials, by the best men to be hired, after the simplest designs that modern engineering can devise. But it will be low in price that no man making a good salary will be unable to own one-and enjoy with his family the blessing of hours of pleasure in God's great open spaces."

-- Henry Ford

"The greatest improvement in the productive powers of labour, and the greater part of the skill, dexterity and judgement with which it is any where directed, or applied, seem to have been the effects of the division of labour."

-- Adam Smith, The Wealth Of Nations

Not surprisingly, our last post on the perils of "One Size Fits All" log management got a heated response from a blogger whose company tilts at the windmills of "mega-SEM" log management. We were called "profoundly stupid," "naive," "incompetent," and "idiotic." We were happy to receive such high praise for our company, which has been producing software in the log management niche since 1997, over twice as long as many of the johnny-come-latelies into the market. Obviously, we're doing something terribly wrong over here :)

Interestingly, the meat of our post, namely that you can put together a good log management system by combining best-of-breed packages that target different types of logs, was not rebutted. Arguably, it is pretty easy to pull some quotes from a blog posting without actually debating the core philosophy or issue. However, we're not really interested in debating this issue, because it would probably devolve into some sort of academic exercise with plenty of jargon and buzzwords that probably don't mean a hill of beans to you, our gentle readers.

One of the most interesting things in the software industry is the disconnect between the "wouldn't it be awesome if?..." theory and the ugly reality of the marketplace. Nowhere is this more painfully obvious than in the area of SEM and log management. In that spirit, and in the spirit of when academia meets reality, we're going to flesh out our previous blog posting into a little thing we call "In Theory and In Reality."

In theory, every possible device, operating system, or program that generates a log would adhere to a common schema or format when doing so. It seems that every year some new working group releases a paper or proposal detailing that very thing.

In reality, only some devices, operating systems, and programs that generate logs adhere to a common format. Cynically or not, vendors of said devices, operating systems, and programs have discovered that there is money to be made selling consulting services and reporting packages for logs written in their proprietary formats. Some of the most popular logging formats, such as Windows EVT files, syslog, and the W3 logging format have gotten that way due to widespread industry adoption and market penetration, not the other way around. On top of that, even if a log is written in a common format, the devil is in the details of the event!

In theory, every organization looking to automate log management has a budget for that project in excess of $50K, or maybe even $100K. On top of that, they obviously would want a log management package that claims to manage hundreds of devices, even though they only have 5 Windows servers, 100 Windows workstations, a UNIX mail server, and a router/firewall on their network.

In reality, many of the admins we work with daily are lucky if their management has blessed them with $5K to spend on log management, never mind $50K.

In theory, most organizations want a large, macro view of logging activity and trends happening across their network. Highly detailed information and reporting would be nice to have, but the big picture is fine for right now.

In reality, if organizations cannot produce detailed, OS/device-specific levels of information for their auditors, they fail audits.

In theory, IT departments are well-staffed with highly-compensated admins who have plenty of free time to spend on extensive consulting and training for the log management packages they adopt. Really!

In reality, IT departments are often poorly-staffed with admins forced into reactive, as opposed to proactive, positions. They need easy-to-configure software that can produce detailed levels of information quickly and without much fuss.

In theory, only expensive, over-engineered SEM packages can produce any useful level of correlation between different devices and operating systems.

In reality, many device and platform-specific SEM packages for SMBs can output aggregated log data into mineable formats such as database tables, or pass that data over the fence to another logging platform (e.g. syslog concentrator, etc), where data can be routinely grepped and mined as needed for key IP addresses, ports, etc.

In theory, all large Fortune 500 companies and huge government entities would naturally want to adopt a mega-SEM package, because it's the only thing that can even come close to dealing with their diverse, heterogenous logging environment.

In reality, many large Fortune 500 companies purchase specific device and platform-targeted log management packages to get a detailed handle on logging data within a certain department. Often, this is after they've been sold a bill of goods by the mega-SEM vendor and they're facing the crunch time of an audit.

We now conclude this chapter of "In Theory and In Reality." We'll soon take you back to your regularly scheduled programming.

The Perils of "One Size Fits All" SEM and Log Management Packages

"Smokey my friend, you're entering a world of pain."
-Walter Sobchak, The Big Lebowski

"A Jack of all trades is a master of none"

Today's post is going be a little outside the technical realm of log management, but is an important post nonetheless.

Often, we receive RFPs (requests for proposal) from companies wanting us to run through a "supports/does not support" checklist of log generating devices. It seems that upper management loves to approach enterprise log management as a quest for the one holy grail product that can manage logs from hundreds of different devices and operating systems, in addition to folding the laundry and making coffee.

This approach to procuring log management technology is fatally flawed from the outset.

The thousands of log generating devices and operating systems in today's marketplace truly and completely prevents any vendor from being a polymath at all of them. Some vendors may try to lay claim to supporting tens, or even a hundred of said devices, but often the reality is empty marketing rhetoric without the robust technology present to deliver on the claims.

For example, the level of nuance and detail in the Microsoft Windows ® event log alone is enough to keep a substantially sized development team busy all the time. We can testify to this, as the Microsoft Windows event log is our area of expertise. Multiply this level of nuance and detail by a factor of hundred, or even a thousand, and you have an untenable goal for even the largest of software corporations.

Moreover, value gets diluted very quickly when you start looking at the price tag of "one size fits all" log management packages, especially when compared to picking up a handful of best-of-breed tools that specialize in log management for specific operating systems or devices. Take a hard look at the reporting in one of those mega-SEM packages and see if that "value dilution" is not readily apparent. 10 to 20 log generating devices may be "supported", but reporting will often be limited to a handful of reports per device.

To play devil's advocate for a minute, let's assume that one of these mega-SEM vendors has a very diligent, hard working development team that cranks out new reports as often as possible. What happens when the way an event gets logged on a particular OS changes or a new service pack is applied? Whoops! Back to the drawing board. Patch, patch, patch and fix all of those previously "finished" reports. As the number of reports increases, each new logging change that happens after an OS upgrade or device firmware patch increases that mega-SEM vendor's work by an order of magnitude. Eventually, entropy will take over, making quality suffer while updates are issued in a less timely fashion. It's a battle that cannot be won, even with the best development efforts and the most earnest intentions.

It's tempting for CIOs and CTOs to buy into the mega-SEM hype - the fantasy of having the logs of hundreds of different devices and computers all neatly aggregated with hundreds of ready-to-be-summoned reports at their fingertips. In fact, one can argue that many of these mega-SEM vendors aren't selling software - they're selling the CxO's dreams right back to them. Unfortunately, these dreams are never fully realized. And the results are tragic:

• Hundreds of thousands, if not millions of dollars, spent on the actual software or appliance


• More hundreds of thousands spent on service contracts and consulting


• Lost employee hours attempting to get the behemoth package to work


• Significant opportunity costs to the business during this process


• Additional software costs when new vendor packages are purchased to produce the sort of information the mega-SEM package was supposed to be delivering in the first place.
  

Enough doom and gloom. Here's a novel philosophy that CxOs can use to reduce the pain and maximize the gain of procuring log management technology:

Step 1: Delegate the work of procuring SEM and log management packages to the department heads that manage the different assets of your network (e.g. the Windows Platform team lead, the *Nix Platform team lead, the Infrastructure/Router/Switch/Firewall team lead).

Step 2: Instruct your various department heads to research and test the best-of-breed log management offerings that are directly relevant to the devices and computers they manage. These department heads are in a unique position to understand the subtle details that can sink or swim a particular SEM package in your environment. They can also tell you the role and quantity of the devices they manage, so you can make a more more targeted distribution of resources (e.g. 80% of all managed devices are Windows servers, 15% are *Nix, and 5% are Other).

Step 3: Empower your department heads to procure the log management package that best suits their realm of your network, and make them responsible for managing, operating, and documenting the software, producing reports on a recurring basis that can be directed to you as needed.

It is our contention that if you adopt this approach, your log management project and procured technology will be:

• Under budget


• Less prone to failure


• Less vulnerable to obsolescense or downtime caused by critical changes in event logging


• More likely to produce higher ROI

Thus we conclude our public service announcement on this topic.

That Infernal Road, Paved With Good Intentions...

Eric, the head auditing guru at Microsoft, posted today on his blog that he is receiving an ever-increasing number of complaints on the lack of documentation regarding the new Event IDs in the Windows Vista™ security log. Specifically, he says that our earlier post "complains" about how sometimes the "add 4096" rule works in Vista's security log, but not in all cases.

With that background, let me take some time here to clarify our original comments and attempt to speak to the source of the frustration Eric is hearing from log management vendors, log scripting enthusiasts, and security admins.

First off, our earlier post on the 4096 offset trick in Vista was not a complaint in so much as it was an attempt to draw attention to a very significant change in the Windows Vista security log. Keep in mind, while Microsoft has made subtle changes to security events ever since Windows NT, the changes in auditing from Windows® NT to Windows 2000 to Windows XP to Windows 2003 are nowhere near as complex as the changes from Windows 2003 to Windows Vista and the forthcoming Windows Server 2008™.

Expanding on this, the complete renumbering of security events in Vista is just the tip of the iceberg. Compounding this trauma of sorts is:

A.) A completely new logging file format, the EVTX file
B.) A completely new API that is used to manage these EVTX files
C.) New, different auditing categories (Tasks) in the Vista security log
D.) Shifting of user account information out of the User field altogether in security events
E.) Other changes to the "traditional" log fields that were present in the legacy EVT files (e.g. the Level/Type field)
F.) Other issues related to forward and reverse compatibility as it relates to log management on pre-Vista and Vista.
... etc

That being said, we know that Eric is not responsible for all of these changes. He did not create the new EVTX log format or the API used to access it, for instance.

Collectively, though, all of these challenges together are most likely frustrating third-party log management vendors, as well as the admins who have developed scripts to automate security event management. Unfortunately, it would appear that Eric is getting the brunt of that frustration. Perhaps he should post contact information for the team at Microsoft that developed the Crimson logging format and accessory APIs so that constructive criticism and questions can be more properly distributed.

At Dorian, our approach is to adapt and innovate around the changes to Microsoft Vista's new logging format and auditing system, and we are proud of our efforts to date. Still, we hear every day the issues that small and medium sized businesses face regarding log management, often directly due to compliance regulations. Not every organization has the budget or resources needed to procure a commercial log management package, and for those facing a complete rearchitecture of their log automation scripts in Windows Vista and Windows Server 2008, those limited resources just got stretched even tighter.

Highlights From the Event Archiver 7 Press Release

Initial feedback on Version 7 of Event Archiver® from customers has been very good. Version 7, in case you didn't read the earlier blog posting, has direct support for Microsoft Windows Vista™ EVTX logs throughout the program. We also added a bunch of cool new features to help overcome some shortcomings in Vista eventing which we are calling LogRefiner™ technology.

This week, we sent out a press release regarding our launch of Event Archiver 7. Here are some highlights, with some of the most interesting sections highlighted in bold:

Dorian Software Creations, Inc. www.doriansoftware.com today announced the release of Event Archiver 7 (www.eventarchiver.com), the latest version of its automated log file collection and consolidation tool.

Having announced earlier in the year a U.S. patent for its Total Event Log Management Solution ™, the globally recognized leader in log management is again charting new territory within the SEM and SIEM markets. This time, Dorian is striking early at the looming onslaught of EVTX files – logs generated by the new Windows Vista and upcoming Windows Server ® 2008 operating systems – that compliance and security specialists face.

Dorian’s development team has been warning for some time in its blog at http://eventlogs.blogspot.com/ that the change in log formats from the existing EVT format to the new EVTX is rife with pitfalls - for admins and particularly, compliance and security specialists seeking consistency and reliability for log audits. The warnings have not articulated a preference between the log types but have instead stressed the importance of understanding the pitfalls before moving forward with Windows Vista and Windows Server 2008 migrations.

Many network administrators and those attempting to audit existing log data have just gotten the hang of the EVT format. Now, within the Windows ®platform alone, these security professionals face the specter of disparate formats and all the problems those differences bring: new event IDs; different formatting of data; and last but not least, changes in the way logs are handled for collection, monitoring, and reporting. Microsoft's shift to the EVTX format in Windows Vista and Windows Server 2008 is truly the elephant in the room for those tasked with ensuring compliance and log retention.

The differences in the log formats and the methodologies behind them are far greater than many in the industry are willing to admit. We are responding to these changes not by forcing upgrades to our software or encouraging adoption of the new format, but by focusing instead on the management of these log types side-by-side. After all, the adoption of the new log format within the private and public sectors is just beginning, and many requirements force organizations to store years-worth of log data. That means, in many cases, auditors and forensic investigators will be looking at the “old” EVT logs for another 5-10 years at least.

...

As a result, Dorian Software Creations, Inc. is introducing its exclusive LogRefiner ™ technology. The focus of this new technology is the careful management of both log formats side-by-side, streamlining the management of both formats via consistent logic and methodology. Therefore, early adopters of Windows Vista and Windows Server 2008 - the operating systems that generate the new EVTX format - can take advantage of log management capability in Event Archiver today. This again sets Dorian Software apart from other log management vendors - almost all of which have been notably mute or at least guarded in their response to the major changes facing SEM and SIEM efforts.

...

Because the management of both log file formats will be necessary for yearsto come, Dorian Software stresses that any releases including the LogRefiner technology will not abandon those who continue to work with the EVT format.

...

Windows Vista EVTX File Support

Event Archiver has the capability to collect and convert EVTX log files. This is the new logging format first introduced in Windows Vista and planned for use in Microsoft Windows Server 2008. Simply install Event Archiver to a Windows
Vista workstation to start collecting EVTX files from other Vista workstations.

LogRefiner ™ Technology Makes Downlevel EVT File Processing in Windows Vista Possible

Dorian's exclusive LogRefiner technology can archive and convert EVT files from downlevel systems directly alongside the EVTX files from Windows Vista and newer operating systems - the converting and reading of EVT files being the very thing that the Microsoft Event Viewer on Windows Vista has difficulty doing correctly. With Event Archiver's special new technology, no information goes missing when converting downlevel EVT files into new formats – all event log fields are processed properly the first time.

Streamlines Fields Between EVT and EVTX Logs With LogRefiner Technology

Did you know that Windows Vista’s EVTX logs have even more fields? Event Archiver 7 can be instructed to automatically consolidate these fields - the Keyword and Opcode fields specifically - into the Task (Category) field so that you can have a uniform data structure for EVT and EVTX exported log files.

LogRefiner Technology Maintains Field Consistency Across
Logs

In the Windows Vista Security Log, no information about the user performing the action or affected by the action is recorded in the User field when an event is logged. Instead, all user information is placed in the Description of the event. Event Archiver 7, however, has the ability to place the most relevant user information back into the User field as it converts EVTX files into new formats. By helping maintain the consistency of log data and its formatting, this feature greatly aids the administrator or compliance officer in charge of reviewing the consolidated data.

Defines Success Audits Versus Failure Audits Using LogRefiner
Technology

Another major change in the Windows Vista security log is that all events are recorded as “Informational.” To discern whether or not the event represents a failed or successful action, the administrator must refer to the Keyword of the event.

But, Event Archiver 7 - when converting security EVTX Files - has the ability to properly record whether or not the event was a Success Audit or Failure Audit, greatly aiding the reviewer of log data generated from both EVT and EVTX log files.

To sum up, our LogRefiner™ technology in Event Archiver 7 means that:

1.) You can migrate to Windows Vista and Windows Server 2008 when you are good and ready, knowing that,
2.) Our software will process the downlevel EVT files for you right alongside the newer EVTX files, and
3.) Event Archiver has advanced technology that standardizes the collected data for reporting and other compliance purposes.

From Windows NT to Windows Server 2008, Event Archiver 7 has you covered. If you'd like to take it for a test drive, you can download your free 30-day evaluation copy at http://www.doriansoft.com/download. Happy archiving!

Storage Requirements for the Windows Vista™ Security Log

Recently, we've created a few blog postings that talk about some of the new events present in the Microsoft Windows Vista™ security log. From a security standpoint, Vista's increased number of auditable events is excellent, as administrators and compliance officers can get a much deeper picture of the actions taking place on a computer prior to and during a security incident.

However, if you are required to retain those security events, either by law (e.g. HIPAA, SOX, GLB, PCI, etc) or by policy, you need to start budgeting for more storage before you start your Vista and Windows Server 2008™ migrations.

Here are a few examples of how Vista security logs tend to grow much more quickly than their predecessors:

1.) Looking at some of our internal Vista security logs, there are tons of events relating to the blocking or accepting of network data via the Windows Filtering Platform. Some organizations may find this data valuable, especially if the machine is exposed to the public, however others may not.

2.) Some events log extra information at the end of the Description field that serves no other purpose than to further explain the parameters in the Description field. For instance, every 4608 event (Windows is starting up) also tells you that:

"This event is logged when the LSASS.exe starts and the auditing subsystem is initialized."

Similarly, every 4634 event (An account was logged off) feels the need to mention that:

"This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer."

These are just two brief examples, but note well: your Vista logs will use up more space than your XP and Windows 2000 workstation logs. If you are reassuring yourself now by thinking that you only need to retain server logs, bear in mind that Windows Server 2008 will share Vista's new events and logging tendencies!

Fortunately, the current release (and several prior releases) of our Event Archiver™ software offers you techniques to help you manage your storage of log data. Event Archiver allows you to automatically prune your database tables by date, selectively import only key events or exclude non-key events into database tables with global import filters, and keep your data in multiple compressed formats for storage efficiency. As the number of auditable events increase and expand in size, these features become increasingly important.

Vista-Compatible Release of Event Archiver is Here!

As promised, we released version 7.0 of Event Archiver yesterday. Event Archiver 7.0 is the first of our log management titles to support Microsoft Windows Vista™ and the new EVTX log format. However, Event Archiver 7.0 is more than just "compatible" with Windows Vista, as it also introduces some very cool abilities our marketing department refers to as LogRefiner™ technology.

Specifically, the biggest LogRefiner™ technology accomplishment is that downlevel EVT files from previous Microsoft Windows® versions get processed correctly when Event Archiver is running on Windows Vista, which the built-in Event Viewer on Vista cannot do properly. Beyond that, it encompasses numerous other features, such as consolidating fields in EVTX files, appropriately categorizing security events as Success Audits and Failure Audits, and placing user information from a Security EVTX file back in the User field. You can read all of the features here:

http://www.doriansoft.com/ourcompany/announcements/6-07.htm

As far as we know, we're the first log management ISV to offer this level of dual EVT/EVTX file processing technology. But, we've also been in the market since 1997, so pioneering new log management techniques is nothing new to us! .

On top of the Windows Vista features, we also added MD5 cryptographic hashing of archived log files and a Working Directory feature for local processing of remote log files.

Needless to say, this is a huge accomplishment that we're very proud of. Now, it's back to the skunkworks to get our other log management titles working with Vista.