Page MenuHomePhabricator
Create Task
Maniphest T206496

Create sitemaps for Indonesian, Portuguese, Punjabi, Dutch, and Korean Wikipedias
Closed, ResolvedPublic
Actions

Description

Background

In T199252: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest we created sitemaps for Italian Wikipedia and measured their effect on traffic from search engines in T202643: Determine if creation of Italian Wikipedia sitemaps increased traffic from search engines. Our results were inconclusive yet pointing towards sitemaps having no effect on search traffic. We would like to see if we can replicate these results on other projects.

Acceptance criteria

  • Generate sitemaps for Indonesian, Portuguese, Punjabi, Dutch, and Korean Wikipedias
  • Submit sitemaps to Google search console
  • (nice to have) add sitemaps to local robots.txt file for other search engines (Not easily possible, can revisit in a separate task if/when we see something to indicate that sitemaps are useful)

Details

SubjectRepoBranchLines +/-
Add sitemaps rewrite for additional domainsoperations/puppetproduction+14 -2
varnish: add vtc test for sitemap rewritesoperations/puppetproduction+33 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

ovasileva created this task.Oct 9 2018, 12:10 AM
Restricted Application added subscribers: revi, Aklapper. · View Herald TranscriptOct 9 2018, 12:10 AM
ovasileva triaged this task as High priority.Oct 9 2018, 12:10 AM
ovasileva added a parent task: T198965: Create XML sitemaps so search engine crawlers can crawl more effectively.
phuedx subscribed.EditedOct 9 2018, 9:52 AM

Our results were inconclusive yet pointing towards sitemaps having no effect on search traffic.

Is this based on T202643#4543393?

• Imarlier added a comment.Oct 9 2018, 3:13 PM

@phuedx - Only in part. Mikhail did a very detailed analysis, based in part on the data that Nuria mentioned there.

• Imarlier changed the task status from Open to Stalled.Oct 9 2018, 3:16 PM
• Imarlier claimed this task.
• Imarlier mentioned this in T206544: Google Search Console access request.

Need SRE to grant me Google Search Console access in order to move forward with this.

gerritbot subscribed.Oct 10 2018, 1:00 AM

Change 465538 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] Add sitemaps rewrite for additional domains

https://gerrit.wikimedia.org/r/465538

gerritbot added a project: Patch-For-Review.Oct 10 2018, 1:00 AM
gerritbot added a comment.Oct 11 2018, 1:29 PM

Change 466602 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: add vtc test for sitemap rewrites

https://gerrit.wikimedia.org/r/466602

gerritbot added a comment.Oct 11 2018, 1:32 PM

Change 466602 merged by Ema:
[operations/puppet@production] varnish: add vtc test for sitemap rewrites

https://gerrit.wikimedia.org/r/466602

• Imarlier added a project: Performance-Team.Oct 11 2018, 7:05 PM
• Imarlier moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.Oct 11 2018, 8:10 PM
• Imarlier added subscribers: • ema, BBlack.Oct 15 2018, 7:33 PM

Blocked on https://gerrit.wikimedia.org/r/c/operations/puppet/+/465538 - @BBlack @ema would one of you be able to help me out?

gerritbot added a comment.Oct 16 2018, 8:14 PM

Change 465538 merged by BBlack:
[operations/puppet@production] Add sitemaps rewrite for additional domains

https://gerrit.wikimedia.org/r/465538

• Imarlier closed this task as Resolved.Oct 30 2018, 9:13 PM

Sitemaps have been generated. I plan to refresh them before submitting to Google, though.

ovasileva added a comment.Nov 6 2018, 7:47 PM

@Imarlier - wondering if you got a chance to submit these yet?

• Imarlier added a comment.Nov 6 2018, 8:49 PM

They're ready to go, just waiting on confirmation from you that they should
be submitted.

ovasileva added a comment.Nov 6 2018, 9:00 PM
In T206496#4726226, @Imarlier wrote:

They're ready to go, just waiting on confirmation from you that they should
be submitted.

Perfect, confirmed.

• Imarlier added a comment.Nov 15 2018, 5:33 PM

Sitemaps for the following sites are now registered with Google:

idwiki
itwiki
kowiki
nlwiki
pawiki
ptwiki

• Imarlier updated the task description. (Show Details)Nov 16 2018, 3:06 PM

FYI: With regards to adding to robots.txt, I don't have the ability to edit MediaWiki:Robots.txt on any of these wikis, so can't make that change without going through a community process. I'm going to suggest that we punt on that for the moment, evaluate specifically the data coming from Google as a result of the Search Console submission, and if it seems like it's worth further testing we can reach out to the communities in order to get those edits done.

• Imarlier updated the task description. (Show Details)Nov 16 2018, 3:25 PM
• Imarlier updated the task description. (Show Details)
mpopov mentioned this in T209720: Determine impact of sitemaps on search traffic to Indonesian, Portuguese, Punjabi, Dutch, and Korean Wikipedias.Nov 16 2018, 5:56 PM
• Niedzielski mentioned this in T208755: Launch A/B test for sameAs property.Nov 16 2018, 6:10 PM
phuedx mentioned this in T223408: Page gets redirected randomly to former blackout page.May 21 2019, 3:39 PM
ovasileva mentioned this in T206497: Enable $wgMFNoindexPages for: Italian, Dutch, Korean, Arabic, Chinese, and Hindi Wikipedias.Jun 21 2019, 5:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits