Auditing and Storage Requirements

One thing that admins tend to overlook when setting up a SEM or event log management package on their network is the amount of storage required to house all of the event log data.

Regardless of which vendor you choose (or even if you decide to attempt to do it in house with scripts), you need to keep in mind that the data output from native Windows event log files (e.g. EVT/EVTX files converted into database tables) will be greater in size than the native event log files themselves.

As mentioned briefly in this earlier post, EVT files contain references to other information not present in the log file itself. The resolution of those references into meaningful data is one reason behind the increase in data size after conversion.

Another factor that causes data size expansion is the field structure of the database itself. The number of bytes allocated for certain fields, as well as the use of UNICODE strings (e.g. two bytes per character) can both further contribute to this phenomenon.

A while ago, we wrote a freeware utility that does a nice job estimating the storage required to maintain your log data in various formats over time. You can download our Auditing Volume Analyzer tool here.

One final caveat: some SEM vendors in the marketplace attempt to take the log storage issue out of the equation by providing their own "black box" or appliance for accumulated log data. In general, we frown upon that approach here as it has the potential to hold your data captive. Should an audit or other incident arise - the last thing you want is data held captive in a proprietary storage system. For more on this topic and our general philosophy on event log management, please read this article.