Log retention best practices

What is log retention?

Log retention, and log retention management, both refer to the storing, handling, and archiving of an organization’s logs — records of events and data that can be used for security analysis, compliance, performance optimization, and debugging. Log retention is part of a broader data retention policy.

The right log retention policies help organizations maintain safe and efficient operations while complying with all applicable laws and regulations. In today’s business environments, an effective log retention management strategy strikes a delicate balance among the organization’s operational, security, legal, and budgetary needs.

And while maintaining compliance and adhering to laws and regulations is a major factor in setting log retention policy, most organizations also hold onto their logs for operational and security reasons as well to improve day to day performance and better investigate security incidents when they occur.

Most business logs fall into the following four categories:

• System logs keep track of system changes, shutdowns, power ups, kernel information, and hardware failures.
• Application logs track and summarize both application behaviors and the behaviors of users while working in those applications.
• Security logs keep track of all events, threats detected, attacks and attack attempts, and logins and login attempts that relate to organizational security.
• Audit logs are logs kept for a prescribed, usually lengthy, amount of time, for auditing, compliance, forensic analysis, and regulatory reasons.

What is a log retention policy?

Log retention policy (otherwise known as log retention management) generally covers the manner in which logs are kept, the logistics of how those logs are stored, the length of time logs are retained, and the accessibility of logs during the storage period.

Although storage capacity has become less expensive over the last several years, there is still a very real cost associated with storing all of an organization’s digital logs for years at a time. So it’s important to have a coherent and realistic policy going forward to ensure that you’re keeping the logs and records you need to be keeping, for the time you need to hold onto them. Otherwise, you’re essentially paying to store more and more logs for more and more time when you don’t actually have to on one hand, or are risking serious fines and penalties as well as reputational damage on the other.

What are the different types of storage and sample log retention periods?

Different types of logs have different storage and retention requirements depending on the sensitivity of the material, its importance, and its business affiliation. For example, a hospital might require different policies for logs pertaining to financial processes versus patient activities. But in general, the following is a good rule of thumb to start with:

• Healthcare: Usually six or more years
• Finance: Usually three to seven years, depending on type and sensitivity
• General: One or more years depending on the industry and log type

Several years of storing millions of records can become quite costly. So companies usually break down the storage (and attendant costs) into three categories in order to most efficiently meet their log storage needs.

• Hot storage is the easiest to access / most readily available. It’s also the most expensive due to the fact that it’s basically always available. Hot storage is used for storing logs and files that your teams might need to access at any moment.
• Warm storage is a little harder to access but less expensive than hot storage, and is more appropriate for logs that you might need access to, but that might take a bit longer to retrieve.
• Cold storage is the hardest to access, the least expensive of the different options, usually takes place off-site, and is essentially the most cost-effective (but least accessible) way to store logs for longer periods of time. Cold storage is often used as another term for archiving.

Why is log retention important?

Log retention is crucial to security investigations, operational analysis, and planned and unplanned audits. While the log retention / security analysis relationship is widely understood, log retention also plays a significant role in examining and fine-tuning an organization’s day-to-day processes, performance, and agility. Having a complete record of your logs will help you maintain compliance with internal policies, industry standards, and government regulations. Today, log retention has morphed from a box to check to maintain compliance into a legitimately business-critical function.

No matter where your organization operates, it’s critical to be in compliance with all local and national laws in the countries where you’re conducting business, so you need to manage all logs and records related to applicable laws like GDPR, SOX, and HIPAA, which we’ll cover in more detail below.

Having a record of your organization’s logs over time is also necessary for investigations and audits. Complying with regulations is non-negotiable, and having an effective log retention policy lays a strong foundation for building an effective compliance practice.

Beyond the importance for logs in compliance, logs can help your security and operations teams see patterns, trends, and issues much clearer than having a few scattered moment-in-time snapshots. Logs can also facilitate forensics. They can help you understand, investigate, and mitigate cybersecurity threats and actual breaches after they happen.

How and why has log retention evolved over the last few years?

Log retention has moved from a simple, passive, usually on-premises process to a more active (and proactive), cloud-based, managed practice with more automated lifecycle management and AI activity incorporated into the workflow.

As business environments become more complicated and increasingly global in nature, modern organizations need a unified approach to log retention and management. The days of data silos and reactive on-premises storage are over. Today, your teams need a centralized, cloud-based, searchable managed solution which keeps you in compliance everywhere you operate, while giving you the access — and the ideal mix of storage capabilities — to suit your specific needs. Increasingly distributed businesses need a common, low-cost repository for all of their logs from all of their locations. The right cloud-based solution solves all of these problems, and in addition to properly retaining your logs, will help facilitate your incident detection, investigation, and remediation efforts as well.

What are the biggest log policy challenges?

• Limited security visibility: A lack of long-term log data makes it much harder for teams to conduct thorough security forensics, detect advanced persistent threats, and understand the full scope of an incident when one occurs.
• Compliance and audit failures: Not retaining certain logs — such as healthcare or payment information-related logs — long enough dramatically increases the risk of non-compliance, legal penalties, and reputational damage.
• High storage costs and complexity: It’s often prohibitively expensive to simply hold on to all of your logs for the foreseeable future. The right log management policy will minimize the high costs and operational overhead of storing massive volumes of log data on-premises or in unmanaged cloud environments.
• Inefficient incident response: With the wrong logs in the wrong storage, you won’t be able to quickly search, analyze, and correlate the logs you need from different sources — and storage levels — when there’s an incident to investigate. This can easily delay both your incident response and troubleshooting efforts.

What are some examples of the regulations involved in log retention and compliance?

While there are literally scores of applicable local and international regulations requiring compliance, here are some of the most influential:

• The Health Insurance Portability And Accountability Act (HIPAA), issued in 1996, concerns patient privacy. It focuses on securing and protecting patient information everywhere it exists, with a focus on portability, as it moves across the US healthcare landscape from patients to and from doctors, hospitals, and insurance systems.
• The General Data Protection Regulation (GDPR), issued in 2016 by the European Union, sets a high bar for privacy standards across all of Europe.
• The Sarbanes-Oxley Act (SOX) is a US act from 2002 that requires all companies doing business in the United States to safeguard their private data, log their electronic records for potential audits, track attempted security breaches, report accurate financial data, and prove they’re in compliance.

Due to the rich mix of personal and payment data, healthcare data is an obvious target for attackers. Not surprisingly, there are a number of strict requirements for storing healthcare logs and data besides HIPAA, along with an industry-specific set of log retention requirements.

Similarly, financial and general business logs are also rich in sensitive private and fiscal data that needs to be protected, and laws like GDPR and SOX, the National Institute of Standards and Technology (NIST) Special Publication 800-53 (NIST SP 800-53), and other regulations lay out the rules to cover privacy protection and the handling and storage of general business and financial logs.

Public sector organizations must also conform to distinct regulations. In the United States, federal, state, and local governments all have their own laws and log retention requirements. For the US Federal Government, stringent laws like FedRAMP have exacting data protection and log retention requirements that all companies doing business with the government must adhere to.

What are the best practices for a state-of-the-art log management solution?

Modern organizations need a proactive, tightly managed, cloud-based solution that combines:

• Real-time log ingestion
• Centralized visibility across the org
• Cost-effective storage solutions

The right solution will help your teams transform your log data, and how it’s managed, into a strategic asset for the entire organization. Other best practices include:

• Creating a formal, written (but always evolving) log management and retention policy
• Listing the various job responsibilities related to it across the org chart
• Centralizing storage, and ensuring the right logs are in the right storage solutions
• Using AI and automation
• Making sure each of your policies start and remain in compliance with all applicable local, national, and global laws and regulations

On a more granular level, you want a solution that can natively ingest your logs and give you immediate access to the intel you need for incident response and real-time analytics through an intuitive, single pane of glass. You’ll also need the ability to customize your log retention periods, as well as your log storage solutions, to meet your organization’s specific security, compliance, and business needs.

How can Cloudflare help streamline log management?

Cloudflare Log Explorer enables you to efficiently store your organization’s logs in the cloud. You can detect security and performance issues, investigate root causes, and mitigate impact — all without adding complexity or cost.

Learn more about how Cloudflare Log Explorer can simplify your log management and enhance your security posture.

FAQs

What is log retention?

Log retention is the storing, handling, and archiving of an organization's logs — records of events and data used for security analysis, compliance, performance optimization, and debugging.

Why is having a log retention policy essential for an organization?

A log retention policy is important for maintaining safe and efficient operations and ensuring compliance with all applicable laws and regulations. It covers the manner, logistics, length, and accessibility of log storage, helping organizations balance operational, security, legal, and budgetary needs.

What are the four primary categories of logs?

Logs might include system logs, application logs, security logs, and audit logs.

What are the three types of log storage?

Log storage is commonly broken down into three categories based on access and cost: hot storage (easily accessible, most expensive), warm storage (less expensive, slightly harder to access), and cold storage or archiving (least expensive, hardest to access).

What are some significant challenges modern organizations face regarding log policy?

Log policy challenges include limited security visibility, compliance and audit failures, high storage costs and complexity, and inefficient incident response.

What are examples of regulations that influence log retention and compliance?

Some influential regulations are the Health Insurance Portability and Accountability Act (HIPAA); the EU’s General Data Protection Regulation (GDPR); and the United States’ Sarbanes-Oxley Act (SOX).

What are some requirements for an effective log management solution?

A state-of-the-art solution should be proactive and tightly managed, combining real-time log ingestion, centralized visibility across the organization, and cost-effective storage solutions. Solutions should facilitate development of a formal log management and retention policy; help centralize storage; use AI and automation; and help streamline compliance with local, national, and global regulations.