eventlogs.com
Event Log Management and the Secure Network
One network professional
recently looked at the amount of event logs being collected daily in their
enterprise and found that 4GB a day were being collected, then compressed
for storage and future retrieval. Company requirements called for the
collection of the log data for auditing purposes. Over time - with weeks,
then months - the compression of that data would be key in managing the
cost associated with meeting this now common requirement.
Many administrators would probably share in the initial surprise at such
small files adding up to such a large amount of data to be collected and
housed. And, this is at the heart of many organizations' failure to act
quickly to collect and store the log data being generated at this very
moment. Because so much of the log files consist of entries recording
the every day details of successful network operation, it is tempting
to underestimate the importance of the data found within the files.
Still, the overwhelming amount of information contained in log files can
be easily managed and stored for the future when that one "needle
in the haystack" may prove to be the difference between understanding
the cause of a future security breach and the details surrounding it.
Automation, then, is the first key in successful event log maintenance
and management. Two factors dictate this: 1) When the archived log files
are called upon, it must be a reliable copy of the data - there can be
no debate as to the integrity of the data itself. As the human element
is removed, the level of reliability increases and the likelihood of finger
pointing is diminished. 2) The overwhelming amount of data; the number
of machines, users, and administrators in the enterprise; and considerations
such as bandwidth and other resources complicate log collection so much
so that an automated solution is the only way to ensure that every event
is collected. How else could one ensure that each and every event has
been successfully collected manually?
One response to the need for automation is to build a scripted, in-house
solution. This is usually offered because of one of the following: 1)
a reluctance to invest resources in a dedicated software solution, 2)
the territorial instincts of network professionals, or 3) a network professional's
desire to create a "nifty" script or to get experience in scripting.
Otherwise, no network administrator would pass up the opportunity to end
the manual collection of logs - the network professionals "in the
trenches" are always the first to see the need for automated collection.
A few very simple factors alone should discourage you from considering the scripted route:
As
security concerns and strategies continue to mature with expanding networks,
storage formats and compression are also key in creating
a successful log maintenance strategy. For smaller networks, Microsoft
Access will serve nicely as an event log database. For growing networks
and large enterprises, Microsoft SQL is the preferred choice as it proves
more reliable in the long run.
One other consideration is that of file type for the event log files.
Recent policies suggest that the native .evt is the preferred format as
it will prove more reliable in forensics work and for law enforcement
purposes.
Of course, the housing in SQL of five years' worth of .evt files from 500 machines is a dreadful thought, but one software solution currently available (see below) can compress log files by 90-95%. Considering compression of the files while doing the initial design of an event log collection strategy will save future headaches and storage costs.
Other resources:
Event Archiver Enterprise
A software for dedicated event log collection,
consolidation, and storage.