[grsec] Kernel Hangs: Highmem and GRSECURITY

[pageexec at freemail.hu]

On 15 Sep 2006 at 0:46, Syed Ahemed wrote:

> I noticed the following based on my month long analysis.
> Since it was evident from /proc/meminfo i was running out of either
> physical or virtual memory during the hang

/proc/slabinfo can also be useful if the leaked memory structure
is allocated in the slab.

> i tried to dig into the source code and found the following code ,
> which after commenting out i no longer see the hang .This is what is
> exactly done in 2.1.9 patch.My question to the community is related to
> this code. Can you throw me some light or pointers where i can find the
> changelog or the reason why this change was made . 

i doubt this chunk can cause the hang directly given that all it
does is to check/enforce the address space size resource limit.
the reason grsec had this chunk at all was a disagreement between
me/spender back then as to how to account for the vma mirrors.

from my point of view, they consume virtual address space, so they
should be accounted, from his point of view they don't consume
extra physical memory so they should not be accounted (the normal
use of the AS limit is to limit physical memory usage by limiting
the size of the virtual mappings, not an exact science even without
vma mirroring).

this is a tricky situation because some apps try/tried to do precise
accounting and set their own AS limit and ran out of it when run under 
SEGMEXEC, so as the easier fix, spender removed the mirrors from AS
size accounting (the proper way would be to fix each affected app to
be not so smart about its own needs, but as you can imagine, that's
quite tedious).

> In the following excerpts of the code the IFDEFINED PAXSEGMEXEC code
> was removed and the kernel hang is no longer seen.

that's interesting, it'd mean that you didn't have a memory leak to
begin with but the AS accounting interfered with the OOM killer (as
in, it didn't kick in in time or something like that).