Paste P6584

(An Untitled Masterwork)

Authored by MoritzMuehlenhoff on Jan 15 2018, 11:09 AM.
1​From: Hugh Dickins <>
2​Date: Thu, 17 Aug 2017 15:00:37 -0700
3​Subject: kaiser: load_new_mm_cr3() let SWITCH_USER_CR3 flush user
5​From: Hugh Dickins <>
8​We have many machines (Westmere, Sandybridge, Ivybridge) supporting
9​PCID but not INVPCID: on these load_new_mm_cr3() simply crashed.
11​Flushing user context inside load_new_mm_cr3() without the use of
12​invpcid is difficult: momentarily switch from kernel to user context
13​and back to do so? I'm not sure whether that can be safely done at
14​all, and would risk polluting user context with kernel internals,
15​and kernel context with stale user externals.
17​Instead, follow the hint in the comment that was there: change
18​X86_CR3_PCID_USER_VAR to be a per-cpu variable, then load_new_mm_cr3()
19​can leave a note in it, for SWITCH_USER_CR3 on return to userspace to
20​flush user context TLB, instead of default X86_CR3_PCID_USER_NOFLUSH.
22​Which works well enough that there's no need to do it this way only
23​when invpcid is unsupported: it's a good alternative to invpcid here.
24​But there's a couple of inlines in asm/tlbflush.h that need to do the
25​same trick, so it's best to localize all this per-cpu business in
26​mm/kaiser.c: moving that part of the initialization from setup_pcid()
27​to kaiser_setup_pcid(); with kaiser_flush_tlb_on_return_to_user() the
28​function for noting an X86_CR3_PCID_USER_FLUSH. And let's keep a
29​KAISER_SHADOW_PGD_OFFSET in there, to avoid the extra OR on exit.
31​I did try to make the feature tests in asm/tlbflush.h more consistent
32​with each other: there seem to be far too many ways of performing such
33​tests, and I don't have a good grasp of their differences. At first
34​I converted them all to be static_cpu_has(): but that proved to be a
35​mistake, as the comment in __native_flush_tlb_single() hints; so then
36​I reversed and made them all this_cpu_has(). Probably all gratuitous
37​change, but that's the way it's working at present.
39​I am slightly bothered by the way non-per-cpu X86_CR3_PCID_KERN_VAR
40​gets re-initialized by each cpu (before and after these changes):
41​no problem when (as usual) all cpus on a machine have the same
42​features, but in principle incorrect. However, my experiment
43​to per-cpu-ify that one did not end well...