Chris Petersen, LogRhythm: The Needles in the Haystack… What Every Security Officer Needs to Know about Log Management
April 2008 by Chris Petersen, founder and CTO of LogRhythm
Totaling up to 25% of an organization’s data, IT logs reveal the security, performance, and status of network devices and applications. Whether or not anyone pays attention, important data on network and security events resides in IT logs. Left unchecked, some of these needles in the haystack can lead to costly outages, security breaches, and loss of sensitive data.
Given the distributed nature of logs, the lack of standardized formats, and the sheer volume of information generated, many organizations have simply ignored this rich datastore of security and operations knowledge. Security and regulatory compliance mandates are making this ostrich approach unfeasible, and driving the need for automated log management to increase network and data security.
Log and Event Management – New appliances hear the tree falling for you
Fortunately for overburdened IT security departments a new class of appliance addresses universal log data collection and analysis. They can perform log collection, log management, archival and restoration, log analysis, event management, and reporting with support for multiple compliance mandates. These products allow delegated administration across functional IT lines and role-based controls so that security, operations, and audit teams have access to only the data and functions they require. With centralized management capabilities they can scale with the growth in log sources and logs generated over time. Here is a summary of the benefits they provide.
Virtually everything on the network – servers, applications, databases, firewalls, switches, routers, POS systems – generates logs. Log and Event Management Appliances can collect the logs via standard protocols such as Syslog and Netflow, and pull logs from Windows hosts and ODBC compliant databases, remote sites, and flat file sources.
Since log formats are as varied as the log sources, the appliance can “normalize” the logs and correlate the timestamps of all log entries to a single ’normal time’ for consistent reporting and analysis without losing the original stamps.
Archival and Restoration
Log and event management appliances can automate the archival and restoration of log data while maintaining the security and integrity of the logs. Based on policies, the appliances maintain a “bookkeeping” data trail. Archived files are cryptographically signed and compressed for tamper proof storage. The restoration process can verify that archives were not modified.
Once collected and normalized, logs are classified and rendered useful to the security, operations, and audit/compliance teams. Logs with immediate relevance such as security events, audit failures, warnings, and errors, then trigger real-time alerts.
The importance of an event can vary by organization, by log source or the impacted asset. The appliance can apply risk-based prioritization based on the:
• Type of event
• Likelihood that the event is real or a false alarm
• Threat rating of the host causing the event (e.g., remote attacker)
• Risk rating of the application, system or device on which the event occurred
Alerting processes can use email, SMS, page, and SNMP, while the user interface can enable quick assessment and drill down to individual log and/or event data for root cause analysis and action.
Log and event management appliances typically offer pre-built reports for specific mandates, including SOX, PCI, FISMA, HIPAA, and others as well as customizable reports.
Automated Log & Event Management – A Must Have for IT Security
The new class of Log and Event Management appliances provide the visibility and synthesized, actionable information from the logs that IT security needs to prevent and head-off insider and outsider attacks. In addition, these appliances help your team meet increasingly demanding audit requirements.