From：Infoworld
Considering how much valuable information is available in log files, you’d think more companies would pay attention to them. Workstations, servers, firewalls, appliances, and other computer devices generate reams of event logs every day, and despite mountains of evidence showing their practical, cost-saving uses, logs often go ignored. A good log management system can help significantly with security, application troubleshooting, compliance, and systems management. If that’s the case — and it is — why do logs and log management sometimes still get a bad rap?
It’s understandable on many levels. First, logs can contain towering amounts of uninteresting, hard-to-decipher events, burying more useful information. In fact, without the appropriate tools and filters, logs can be nothing but noise — and lots of it.
A standard Microsoft Windows computer can easily generate thousands of events each day even when things are humming along without a real problem. A thousand computers can generate tens of gigabytes of log files on a daily basis. I’ve seen enterprise event log collector tools bring robust networks to their knees. What’s worse, many administrators would tell you that in a typical week, not a single issue requiring an immediate response was uncovered. “Talk about a waste of resources,” they will tell you, even as valuable, useful data is passing under their eyes.
Diamonds in the rough
Log file review is rarely a management priority — until it hits a tipping point or the auditors complain loud enough. Many studies have shown that the majority of security events and application errors would have been noticed earlier had the relevant log files been reviewed. Yet management tends to act as if logs aren’t worth the time and effort to analyze, a dismissal that trickles down to overworked staff. Why mess with something that seems like a waste of time to all parties involved?
Another factor is simply human nature: Few people get excited about reviewing log files. The answer to “Hey, Johnny, what do you want to be when you grow up?” is never log file reviewer, even if a good log reviewer is actually worth his or her weight in gold.
So why should you or your company care about log files? Because they allow an IT organization to be proactive versus reactive. The typical IT department waits for calls for help before responding to problems. But by the time end-users call in, they are already frustrated, the event that prompted the call has typically entered a critical phase, and IT is forced to respond in the most inefficient manner possible.
Imagine how delighted your end-users would be if the help desk called them ahead of time to let them know they were having a hardware or software problem that was just starting to manifest itself. Wouldn’t it be nice to catch hackers before they were successful? Can you imagine a world in which your purchasing department was alerted to buy additional hard drives before they ran out of free space?
Are log files a waste of time? The exact opposite is true. Logging, if appropriately configured and managed, will save you and your company time and money. The best-run organizations live on a diet of event log baselines and proactive responses, and you can too.
Log management 101
In a nutshell, logging allows you to quantitatively and proactively measure the overall health of your environment, from a security perspective, for auditing and compliance, for systems management, and for application tuning and troubleshooting. These basics will get you started.
Security monitoring. Most of the literature surrounding computer logging talks about monitoring events to lower your security risk. Logging can alert incident response teams to prevent malicious hacking in the first place — or at least send in the cavalry as quickly as possible after an exploitative event has occurred to minimize damage and start foren sic investigations.
Logging security events for intrusion detection and forensics, which is often the main reason administrators get into log management, requires specialized advice. You can start by reading NIST’s Special Publication 800-92, “Guide to Computer Security Log Management.” Released in September 2006, it’s unusually easy to read for a NIST (National Institute of Standards and Technology) publication and extremely useful for deploying event log management systems in the real world. It’s considered the gospel in this small corner of the computer security world.
The NIST guide steps through all of the essentials of log file management: identifying the threats and risks to your environment; determining policies for logging, auditing, and handling logs; collating, indexing, and normalizing logs for analysis; defining and generating alerts and actions for critical events; and defining reports and metrics for management review. From putting log management infrastructure and processes into place to reviewing and archiving logs, it leaves no stone unturned.
Auditing and compliance. As motives for instituting log management programs, auditing and compliance are becoming as important as traditional security requirements. Most industry regulatory guidelines now define specific security events that must be monitored. When the right audit policy has been enabled across all required computers, and the appropriate log management system is in place, most companies will pass that portion of a compliance review. On the other hand, the lack of an acceptable security auditing policy can raise suspicion that the right controls are lacking, which may have legal implications.
Systems management. The best-run shops understand the value of logging and use it for systems management. These organizations create baselines of normal operating activity and events, and they set up alerts triggered by excessive deviations. Many environments execute simple ping connectivity tests to monitor which devices are online and which unexpectedly dropped and need to be investigated. Other places embrace the full richness that logs provide.
If a hard drive begins to move too many bad sectors, even before a complete crash occurs, the log administrator has a replacement drive ready to roll. If network activity spikes unexpectedly, administrators are aware of the problem before the inevitable complaints about slowness arrive. A sustained traffic hit may be a worm or a denial-of-service attack. If a server or SAN crashes, the help desk knows about it before users start to call in.
