Page MenuHomePhabricator

Create sitemaps for Indonesian, Portuguese, Punjabi, Dutch, and Korean Wikipedias
Closed, ResolvedPublic

Description

Background

In T199252: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest we created sitemaps for Italian Wikipedia and measured their effect on traffic from search engines in T202643: Determine if creation of Italian Wikipedia sitemaps increased traffic from search engines. Our results were inconclusive yet pointing towards sitemaps having no effect on search traffic. We would like to see if we can replicate these results on other projects.

Acceptance criteria

  • Generate sitemaps for Indonesian, Portuguese, Punjabi, Dutch, and Korean Wikipedias
  • Submit sitemaps to Google search console
  • (nice to have) add sitemaps to local robots.txt file for other search engines (Not easily possible, can revisit in a separate task if/when we see something to indicate that sitemaps are useful)

Details

Related Gerrit Patches:
operations/puppet : productionAdd sitemaps rewrite for additional domains
operations/puppet : productionvarnish: add vtc test for sitemap rewrites

Event Timeline

Restricted Application added subscribers: revi, Aklapper. · View Herald TranscriptOct 9 2018, 12:10 AM
phuedx added a subscriber: phuedx.EditedOct 9 2018, 9:52 AM

Our results were inconclusive yet pointing towards sitemaps having no effect on search traffic.

Is this based on T202643#4543393?

@phuedx - Only in part. Mikhail did a very detailed analysis, based in part on the data that Nuria mentioned there.

Imarlier changed the task status from Open to Stalled.Oct 9 2018, 3:16 PM
Imarlier claimed this task.

Need SRE to grant me Google Search Console access in order to move forward with this.

Change 465538 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] Add sitemaps rewrite for additional domains

https://gerrit.wikimedia.org/r/465538

Change 466602 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: add vtc test for sitemap rewrites

https://gerrit.wikimedia.org/r/466602

Change 466602 merged by Ema:
[operations/puppet@production] varnish: add vtc test for sitemap rewrites

https://gerrit.wikimedia.org/r/466602

Imarlier moved this task from Inbox to Doing on the Performance-Team board.Oct 11 2018, 8:10 PM

Change 465538 merged by BBlack:
[operations/puppet@production] Add sitemaps rewrite for additional domains

https://gerrit.wikimedia.org/r/465538

Imarlier closed this task as Resolved.Oct 30 2018, 9:13 PM

Sitemaps have been generated. I plan to refresh them before submitting to Google, though.

@Imarlier - wondering if you got a chance to submit these yet?

They're ready to go, just waiting on confirmation from you that they should
be submitted.

They're ready to go, just waiting on confirmation from you that they should
be submitted.

Perfect, confirmed.

Sitemaps for the following sites are now registered with Google:

idwiki
itwiki
kowiki
nlwiki
pawiki
ptwiki

Imarlier updated the task description. (Show Details)Nov 16 2018, 3:06 PM

FYI: With regards to adding to robots.txt, I don't have the ability to edit MediaWiki:Robots.txt on any of these wikis, so can't make that change without going through a community process. I'm going to suggest that we punt on that for the moment, evaluate specifically the data coming from Google as a result of the Search Console submission, and if it seems like it's worth further testing we can reach out to the communities in order to get those edits done.

Imarlier updated the task description. (Show Details)Nov 16 2018, 3:25 PM
Imarlier updated the task description. (Show Details)