From Six to Two

It has been reported this week that upstream Linux maintainers are ending their eXtended Long-Term Support (XLTS) experiment. The creation of XLTS was first announced unexpectedly six years ago, in September of 2017. With the sunsetting of this experiment, understandably due to maintainer burnout, upstream LTS kernels will eventually return to their previous pre-2017 two years of support. In this post, we'll discuss why this move will not affect customers of our grsecurity stable kernel offering, and how security-conscious businesses can benefit from switching to our minimum three-year supported stable offering.

Current upstream-maintained LTS kernels list their EOL dates on https://kernel.org/category/releases.html :

As shown, the last remaining 4.x LTS will reach EOL upstream next year, while the remaining existing LTS will all reach EOL by the end of 2026. As we've noted on other platforms in the past, however, an unexpected reduction in the standard of six years of support had already occurred with the 5.15 LTS and later the 6.1 LTS. Based on their release and EOL dates, 5.15 will receive only five years of updates upstream, while 6.1 will receive only four.

The above six upstream LTS kernels are offered under a large caveat described by Linux kernel stable maintainer Greg KH on his blog:

The problem with this caveat is that at some point every LTS kernel offered upstream will eventually become one of these "older releases" missing even critical security fixes with CVEs assigned. The point at which that happens is unfortunately never communicated to upstream LTS users, even if it occurs as soon as three years into the LTS cycle. This creates uncertainty that product and ops teams are unable to prepare for.

Improving upon upstream's LTS offerings, grsecurity's supported LTS kernels do include these fixes and are fully intended for general purpose systems and other high-security-need environments.

We know how hard it is for volunteers to keep up with backporting fixes to so many different releases, especially for code that is developed by others. It's for this exact reason that we've always resisted pressure to support more kernel versions than we do, or for longer periods of time, because we want to keep quality consistent. That same motivation has been behind grsecurity's independent process for selecting and performing backports, focused on security and reliability issues.

Get Up to Speed on grsecurity

grsecurity has existed for 22 years, from the very first 2.4 Linux kernel to the very latest 6.x and every version in-between. If this is the first you've heard of us, here's a recap of what we offer:

Proactive 0-day/N-day Defense

We offer advanced security defenses against entire classes of vulnerabilities and exploitation techniques. It's generally the case that in the event of a published Linux kernel exploit that the exploit is prevented by not just one of grsecurity's features, but individually by several different features.

We invest heavily in both offensive and defensive research, the results of which end up informing approaches in not only later versions of Linux, but all other modern OSes. Our pedigree includes creating Address Space Layout Randomization (ASLR), the full and permanent separation of code and data (otherwise sometimes called "strict" W^X), functionality that later became part of x86/ARM hardware via SMEP/SMAP/PXN/PAN CPU features, the first production-grade kernel and hypervisor Control Flow Integrity system, and much more.

One of our main areas of innovation for the past decade has been through augmenting the GCC compiler through compiler plugins to provide novel and effective security properties for our customers (for instance, to provide a much more comprehensive and scalable approach to Spectre vulnerabilities than the manual ad-hoc approach applied upstream). These defenses are critical to proactive security and the prevention of 0-day and N-day vulnerabilities.

Independent Security/Stability Backport Process

As mentioned above, we provide the results of our security and stability-focused independent backport process for the stable kernel versions we support. Our download page readily shows the kernel versions we support and their associated EOL dates:

Updates are made available every few days. When new upstream LTS releases are published, grsecurity is generally updated to that version the same day and updates published once internal testing is complete (more on that testing later).

Our LTS kernels have always been supported for a minimum of three years. More recently, we are also selecting a newer LTS kernel each year and supporting it for a period of a year (we call this "short-term stable"). Each of these kernels receives the same attention as far as backports and grsecurity-specific bugfixes and enhancements.

Our outgoing stable kernels also generally continue to be supported for a few months past their official EOL date in order to ease the transition for customers to a newer stable version. We are always available to assist with any issues encountered during these version updates.

Expert Extension of Kernel/Security Teams

We provide swift and accurate guidance, bugfixes, and custom development for our customers. Unlike some vendors that may only support the specific binary provided to their customers, grsecurity is designed to be customizable to the customer's specific needs. Customers are able to make full use of the GPL-licensed source code of grsecurity, and via our Pro Support offering, we can adapt any third-party kernel functionality both for compatibility but also so that it can take full advantage of the added grsecurity-provided security features. On our download page for customers are patches and automated self-patching installers for third-party software like AUFS, VMware Workstation, NVIDIA video card drivers, OpenZFS, and Virtualbox.

All of our developers and security researchers are also highly-experienced reverse engineers. This is especially important when a customer encounters an issue, as even in the common case where we are provided with minimal preliminary information, we can root-cause the issue and provide a quick resolution.

The lengths we go to in resolving issues for our customers is practically unheard of in the rest of the industry -- even if the issue ends up being caused by a bug in hardware. We blogged about one such example involving some Intel Atom CPUs back in 2021.

Stable Process Differentiation

Over the years, we've noticed and reported on process issues with upstream LTS maintenance, many of which have unfortunately not been resolved to date. For the interested reader, slides 20-26 of my presentation at the 2020 Linux Security Summit summarizes some of these findings. These continued findings informed our own processes, some of which we describe below:

Consistent Security

Our extensive use of plugin-based compiler enhancements for security functionality both eases the maintenance burden of advanced security functionality and mitigations for some of the more challenging CPU vulnerabilities. This makes that functionality immediately usable by all customers regardless of their particular GCC compiler version. Furthermore, our consistent backporting of our own security advances is in stark contrast to upstream LTS processes, which often do not involve backports of security hardening critical to thwarting exploitation of 0-day and N-day vulnerabilities.

For instance, the upstream 5.15 LTS scheduled to be EOL'd near the end of 2026 has no support for Intel CET, a requirement for hardware-assisted Control Flow Integrity (the security feature that prevents ROP attacks in the kernel), whereas in grsecurity we have production-grade compiler-based CFI for all of our supported stable kernels, including 5.15 and our older 5.4 stable kernel, and have been able to offer this capability for the past eight years already.

Extensive Testing

We have multiple separate automated testing environments with different goals. Kernels receive compile, boot, and regression testing against a variety of artificial configurations to test corner cases across all of our supported architectures, as well the base kernel configurations used for each major version of all popular modern Linux distributions.

24/7/365 customized syzkaller fuzzing is performed against all kernel versions we support. LTP, kselftests, and fstests are run daily to spot further issues before they impact customers. An automated in-house binary analysis system using professional disassemblers/decompilers continuously verifies security properties at the binary level, as a guard against relevant errors in the 'objtool' tool used by the Linux kernel.

Most importantly, the results of all this testing are constantly scrutinized and acted upon, as any failures or new issues found are immediately brought to the attention of our developers.

Independent Patch Review

Our backport process begins by manually inspecting upstream commits and flagging ones with security/reliability relevance, irrespective of any CVE or other public vulnerability announcement. This manual process is supported by an extensive automated system that provides safeguards to the process, ensuring for instance that when a fix is backported, it is not already known to be bad. The automated system also ingests any available information about when a bug was introduced (either in mainline or via a later LTS backport) to reduce in many cases the amount of time required to determine what kernels are affected.

This process ensures fixes for unpublicized (whether intentional or not) security/reliability issues land in grsecurity weeks or months prior to their potential appearance in Linux distribution kernels.

Decades of Wide-Ranging Experience

Our team, given its extensive experience with all Linux kernel versions for the past 20+ years, is knowledgeable and experienced across all major subsystems of the kernel. Under the upstream LTS process, a difficult backport of a CPU vulnerability mitigation may depend on volunteer work of an already-stretched-thin maintainer. Thus, it's no surprise that for instance the upstream 5.4 LTS, despite claims of support for more than two years from today, still contains no fixes for the AMD Retbleed CPU vulnerability announced last year or the AMD Inception vulnerability announced earlier this year. Customers naturally expect these important security issues to be fixed, and our experience, expertise, and knowledge of changes in the kernel over time, enable us to fix these issues correctly and verify their correctness.

It is part of our standard process for published vulnerabilities or exploits to analyze the issue, defeat published techniques, and identify any ancillary techniques or undiscovered issues. As just one example, when one of our security researchers performed this analysis against the Retbleed vulnerability last year, he discovered and reported yet another AMD CPU vulnerability.

Previous users of grsecurity stable kernels and our current customers can attest to the importance of our independent backport process, but the results also speak for themselves. In March of 2020, we reported that at the time of our 4.4 stable kernel EOL, it contained over 1250 security-relevant fixes missing from the upstream 4.4 LTS kernel at the time, which was scheduled to be supported in some form for an additional two years.

Stability Among Instability

In summary, as upstream Linux winds back down its LTS support from six years to two, grsecurity and its customers will remain unaffected and benefit from a continued minimum of three years of support, giving product and ops teams more time to focus on what matters to their business.

Businesses with high security requirements who also either prefer or due to vendor requirements must stay on a particular major Linux kernel version are invited to contact us at contact@grsecurity.net to discuss their needs.