New pages with apostrophe characters aren't indexed by external web search engines. Sampling a random sentence from a page with a title containing ' character in its title will yield no result in web search. The examples below aren't too fresh - so it is old enough for search engines to index.
Old pages (0.5 year old) containing apostrophe character in their title are indexed - so this is probably a regression from the last month or so. I'm not sure whether it is a MediaWiki issue or search engine issue (cross posts - see below).
Exmples
English Wikipedia examples:
- https://en.wikipedia.org/wiki/Conference_USA_Men%27s_Soccer_Freshman_of_the_Year (works in Bing, doesn't work in Google and Baidu)
- https://en.wikipedia.org/wiki/Holland%27s_Next_Top_Model_(cycle_8) (works in Bing, doesn't work in Google and Baidu)
Hebrew Wikipedia examples:
- https://he.wikipedia.org/wiki/%D7%9E%D7%99%D7%A7%D7%94_%D7%91%D7%96'%D7%96'%D7%99%D7%A0%D7%A1%D7%A7%D7%99 (created at 30 August 2015; doesn't yield results in Google and Bing)
- https://he.wikipedia.org/wiki/%D7%92'%D7%A8%D7%9E%D7%99_%D7%A7%D7%95%D7%A8%D7%91%D7%99%D7%9F (created at 14 August 2015; doesn't yield results in Bing and Google)
Debug info
I"m not sure it is a MW issue but I tried to check the following options
- probably not an issue with dumps script - e.g bzcat DUMPFILE | grep _NEW_PAGE_WITH_APOSTRPHE
- probably not an issue with API (there are no complaints on other issues that this would cause)
- probably not an issue with new pages ATOM
- no relevant exclusion in robots.txt (if I didn't miss anything)
See also