Graphs show us that PrefetchingPropertyTermLookup::prefetchTerms() is being called around 70K times every minute. This is clearly too much for something like properties. I suggest adding a layer of APCu cache with TTL of ten seconds to keep the hot data in the servers for long period of time without drowning it.
We dove a bit deeper into all of the things that are happening surrounding the prefetching, term buffers, lookups, fallbacks and caching and decided that to do this properly we should probably aim for something like this:
This doesnt really account for just wanting to retrieve raw terms (rarther than term fallbacks yet), but that could also be worked in, even with its own cache.
This topic area might be a good candidate for a Wikidata-Trailblazing-Exploration
So I noticed that this change can now be seen in some other metrics now too (once I finally narrowed 12th 13th 14th of feb 2020 down to 1.35.0-wmf.19 and finally this ticket and these commits.
Below is the grafana dashboard showing the time it takes to generate parser output for Wikibase items.
At the time this change was deploy we saw median "parse" time (and also p75 parse time) for Wikibase item halfed (~200ms -> ~100ms)
(which is awesome)