Currently, OutputPageBeforeHTMLHookHandler always fetches an EntityRevision for displaying the termbox. Entity revisions are cached by CachingEntityRevisionLookup, so this does not hit us as hard as it could in terms of wgBackendResponseTime. The memory footprint due to that caching is probably quite high, though.
For entity page views on uncached entity revisions, getting rid of this should half wgBackendResponseTime.