[Feature] Better detection of LSM availability errors and proper fallback

[aerosouund]

What problem does this solve?

We currently detect BPF availability in the project through seeing if there is a bpf entry in /sys/kernel/security/lsm. But that file may have a false entry, or may not be mounted on that path. Resulting in a state where LSM may be present but undetected or falsely announced as available. We currently fallback to fmod_ret for all of those cases. A better strategy to detect this would be to actually try to load and attach LSM programs during init, and if they fail only do we fallback to trying fmod_ret which may also fail. In that case open/exec enforcement becomes truly unavailable and this should be announced to the user

Proposed change

See the fallback review discussion

Alternatives considered

No response.

Additional context

No response.

Delivery planning

Outcome: Trust protection. Priority: P1.

Attempt actual BPF-LSM load/attach, clean up failed partial state, then try fmod_ret. Test missing securityfs, load failure, attach failure, fallback success, and both paths unavailable. Record the selected hook and failure reasons per affected policy/node. Coordinate status with #188; behavior checks belong in #191.

Follow the repository build/test, kind behavior, documentation and redaction requirements. Acceptance must distinguish prevention from observation.

[Anujramola1999]

Context:

The finding: "attach succeeded" does not mean "the hook runs"

I loaded the daemon with lsmEnabled forced to true (i.e. skipping the /sys/kernel/security/lsm check, which is what this issue proposes) on an Ubuntu 24.04 VM (kernel 6.8.0-134-generic, aarch64, limactl, k3s v1.36.4) where bpf is not in the active LSM list:

$ cat /sys/kernel/security/lsm
lockdown,capability,landlock,yama,apparmor

Distributions like Ubuntu ship a kernel with CONFIG_BPF_LSM=y. That is enough for the LSM hook stubs (bpf_lsm_file_open, bpf_lsm_bprm_check_security) to exist in BTF, so the daemon can load and attach the LSM programs without any error. But whether the kernel ever calls those stubs is decided separately at boot by the lsm= list — and bpf was never enabled there. The programs attach and sit on the hook; the kernel routes every security_file_open past them without ever invoking them.

Attach succeeded, verified at the kernel:

$ sudo bpftool link list
49: tracing  prog 1063   prog_type lsm  attach_type lsm_mac
50: tracing  prog 1065   prog_type lsm  attach_type lsm_mac

But the programs never ran. With kernel.bpf_stats_enabled=1, after 200 file opens and 200 execs on the host, bpftool prog show for 1063/1065 reports no run_cnt at all. As a positive control, the daemon's own trace_exec raw tracepoint went run_cnt 502 → 708 in the same window, so the counter itself works.

The policy still reported success. An enforce RuntimePolicy with open.deny: [/etc/passwd] and exec.deny: [/usr/bin/curl] selecting a curlimages/curl pod showed EnforcementAvailable=True, PodsMatched=True, Applied=True reason: Enforcing.

And it enforced nothing:

$ kubectl exec lsm-repro -- cat /etc/passwd
root:x:0:0:root:/root:/bin/sh      # exit 0
$ kubectl exec lsm-repro -- curl --version
curl 8.10.1 (aarch64-unknown-linux-musl) ...   # exit 0

Control, same pod, same policy, unpatched daemon (reads the file → false → fmod_ret on the same security_file_open hook): run_cnt climbing continuously, and both actions blocked (Operation not permitted, exit 1 / 255).

This is the well-known "ghost attach" scenario: CONFIG_BPF_LSM=y makes attach succeed, an lsm= list without bpf makes execution never happen, and nothing at attach time tells the two apart. Two consequences:

1. The current /sys/kernel/security/lsm check is what is protecting stock Ubuntu nodes (and AKS default / GKE Ubuntu pools) today. "Try LSM, fall back only if attach fails", as the issue body proposes, would select LSM on those nodes and silently enforce nothing.
2. docs/users/platforms.md and the header comment in ci.yml both say the kernel "refuses the LSM attach" without bpf in the list. On this kernel it does not refuse; it accepts and ignores.

The plan

The kernel keeps a counter of how many times each BPF program has run (run_cnt). So instead of trusting the file or the attach result, the daemon can simply check that counter.

At startup the daemon will:

1. Attach the hooks (LSM or fmod_ret, whichever the file suggests first).
2. Do one harmless open and one harmless exec of its own binary.
3. Look at the counter.
   • It went up → the hook works, keep it.
   • It stayed at zero → ghost attach. Detach, and try the other hook type the same way.
4. If neither type works, report open/exec enforcement as unavailable instead of pretending it works.

The counter is switched on only for this check and switched off right after. The whole thing takes about half a second, once, at startup.

I have implemented this and tested it on the same VM:

• Normal start: picks fmod_ret, confirms it runs. Same as today, but verified.
• Forced to prefer LSM (the ghost case from above): the daemon now logs
  open/exec hooks attached but never executed, drops the LSM hooks and switches to fmod_ret.
  The same cat /etc/passwd and curl --version that went through before are now blocked.
• Added a test that attaches the real LSM and fmod_ret hooks and checks the counter. Needs root, but no special boot parameter — on a plain Ubuntu machine it catches the ghost and passes.
• On old kernels without the counter (before 5.8) nothing changes from today; the daemon just logs that it could not verify.
• Fixed the wrong sentence in docs/users/platforms.md and ci.yml ("the kernel refuses the attach" — it does not).

Not tested by me: a node where BPF-LSM is really enabled (should keep LSM), and a kernel without CONFIG_BPF_LSM (should refuse the attach and use fmod_ret). Both are covered by unit tests; a real AL2023 or Azure Linux node would confirm them end to end.

The implementation is in #251.