Lab · Systems
Free memory looks fine. The machine is thrashing.
Your alert watches free bytes, and it did not fire during the last incident. That is not a tuning problem. Drag the threshold across three workloads and try to find a value that handles all of them.
- free
- Never fired
- psi
- Acted at 15s
The working set is a little larger than what fits, so reclaim runs continuously and every page it drops is wanted again seconds later. Free memory never looks alarming — reclaim is keeping it healthy, and that is exactly the problem. The cost shows up as stall time, and only one of these two signals is measuring it.
- free
- Late by 6s
- psi
- Acted at 18s
A large app launches and takes the budget down with it. This is the case free bytes is actually built for, and with a high enough threshold it does fine — note when it fires versus when PSI does. Keep this scenario in mind while you raise the threshold to catch the first one.
- free
- Stayed quiet
- psi
- Stayed quiet
Ordinary app switching. Free memory swings hard and often — that is the page cache doing its job, not distress — and the system is never in trouble. Every time a policy fires here it evicts something warm for nothing, and the user pays for it on their next launch. This is the scenario that punishes a high threshold.
Free bytes @ 100 MB
1 missed · 1 late · 0 false alarms
PSI @ some avg10 > 20%
0 missed · 0 late · 0 false alarms
Swept every threshold from 0 to 400 MB: not one of them handles all three workloads. Low enough to leave healthy churn alone is too low to catch the refault storm; high enough to catch the storm evicts warm apps all through the churn. The knob is not mistuned — it is reading the wrong quantity.
PSI does it with one number because it is measuring a different kind of thing. Free bytes is a level, and a level says nothing about what it cost to hold. PSI is time — the share of the last ten seconds in which some task sat waiting on memory. That is the same stall the user feels, which is why it does not need retuning per workload.
How to read it
The shaded band is ground truth: the window in which tasks are actually stalling and the user is actually paying for it. It is defined by the harm, not by either policy, so neither signal gets to mark its own homework. Each policy draws a vertical line where it fires.
The refault storm is the case that matters. The working set is slightly too large, so the kernel reclaims continuously and immediately reads back the pages it just dropped. Free memory sits at a comfortable 280 MB the entire time — not despite the thrashing but because of it. Reclaim is succeeding. That is what a healthy free-memory figure means: the kernel is keeping up. It says nothing whatsoever about what keeping up is costing.
So raise the threshold until it catches that, and look at healthy churn. Free memory swings between 178 and 322 MB there because the page cache is doing its job, and the system is never in trouble. Every firing in that panel evicts a warm app for nothing and charges the user a cold start when they switch back.
Why no threshold works
Free bytes is a level. Pressure is a rate. A level tells you where you ended up; it cannot tell you how hard the machine worked to keep you there, and the work is the part that hurts. Two systems reporting an identical 280 MB free can be in completely different states — one idle, one refaulting its working set several times a second — and no threshold on that number can separate them, because the number is the same.
Reaching for MemAvailable instead of MemFree does not fix this. It is a better estimate, and it is still a level: it predicts how much you could reclaim, not what reclaiming is already costing.
PSI measures the cost directly. From /proc/pressure/memory, or per-cgroup at /sys/fs/cgroup/<slice>/memory.pressure:
some avg10=47.21 avg60=31.08 avg300=12.44 total=8419283
full avg10=11.902 avg60=6.31 avg300=2.07 total=1904772some is the share of the last ten seconds in which at least one runnable task was stalled waiting on memory. full is the share in which every non-idle task was stalled — by the time that one is high you are not degraded, you are stopped. Both are wall time lost, which is the same quantity the user experiences, and that is why one threshold survives all three workloads without retuning.
What to do with it
Alert on some avg10 for early warning and full avg10 for “we are already down”. Keep the free-bytes alert if you like — it is a fine last-resort backstop — but stop treating it as the detector. It needs Linux 4.20 or newer, and cgroup v2 for the per-cgroup figures.
This is not a niche opinion. systemd-oomd, Meta’s oomd, and Android’s lmkd all moved their kill decisions onto PSI rather than free-memory watermarks, for the reason the third panel shows: watermark-based killing either fires too late or fires constantly, and there is no setting in between.
The longer argument — including why this became the observation space for a reinforcement-learning memory policy rather than a hand-tuned heuristic — is in Reinforcement learning over cgroup v2. The three workloads here are constructed to isolate one behaviour each. They are illustrations of a mechanism, not benchmark results.