[grsec] Pax killing off Xorg, even with chpax -spmr and /dev/{mem, kmem} restrictions off

[fire-eyes]

I'm using grsecurity with kernel 2.6.7, with gradm 2.0.1. It keeps
catching Xorg, even though i chpax -spmr Xorg bin, ideas? 

This is what I see:

PAX: execution attempt in: <anonymous mapping>, 081f4000-08254000
081f4000
PAX: terminating task: /usr/X11R6/bin/Xorg(X):7129, uid/euid: 1007/0,
PC: 08251d48, SP: 5f7974dc

What bothers me is that this all worked fine at work, on a system with
the same distro etc.