See also discussion on Twitter; various other search systems also don’t find all Q-pages.

https://twitter.com/danbri/status/1362640655399473155?s=21

Quick update: @dr0ptp4kt is helping with getting the needed info/access to Google Search Console in T275240 so we can have a closer look and understand what's going on.

Ok thanks to @dr0ptp4kt I now have access to the magic search console \o/

Results so far: https://www.wikidata.org/wiki/Q99922367 was indeed not indexed at all. From a quick look of it because nothing links to it. I've pressed the button to get this one page indexed but that of course doesn't solve the underlying problem nor does it help with search engines other than Google.

Next steps: I think someone who knows more about all this need to walk through this data with me. I want to make sure I'm interpreting it right before thinking about potential solutions.

See also T200846#4487680.

How about generating sitemap files during munging (rather than as part of
mediawiki or wikibase frontend)?

Quick update: @dr0ptp4kt and I talked about this some more today and I did a bit more digging. One thing I now notice is that Q99922367 isn't linked to from anywhere even within Wikidata: https://www.wikidata.org/wiki/Special:WhatLinksHere/Q99922367 Is that maybe a low-key thing we can try to fix, even without a sitemap?

Will continue discussions with @dr0ptp4kt.

In T273113#6890378, @DanBri wrote:

How about generating sitemap files during munging (rather than as part of
mediawiki or wikibase frontend)?

Sounds possible, but also sounds massive.
I guess this would just end up being a 100 million line text file that constantly grows.