Page MenuHomePhabricator

Very long response time on frwiki main page
Closed, ResolvedPublic

Description

Hello, there is a very long response time on frwiki main page (https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal).

When I'm logged on, it takes around 20 seconds to receive the first byte from the server.

This issue doesn't appear when I'm logged out.

Event Timeline

Shawn created this task.Fri, Oct 30, 12:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Oct 30, 12:22 PM
Aklapper triaged this task as High priority.Fri, Oct 30, 1:04 PM
Aklapper added a project: Traffic.

I can confirm that from Europe, both in Firefox and Chromium.

Restricted Application added a project: Operations. · View Herald TranscriptFri, Oct 30, 1:04 PM

(In theory I'd point to https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue but I don't see which of those items might be helpful here.)

CDanis edited projects, added serviceops; removed netops.Fri, Oct 30, 5:51 PM
CDanis added a subscriber: CDanis.

This isn't limited to just esams; it is in fact happening across all cache clusters.

All of my requests took at least 19 seconds. Some of them took ~38 seconds.

Additionally you don't need to be logged in to reproduce this; other ways that produce a cache miss also trigger this behavior.

I will hazard a guess: something was changed recently that causes rendering the frwiki front page to trigger an RPC guarded by Poolcounter. Said RPC times out.

jijiki raised the priority of this task from High to Unbreak Now!.Fri, Oct 30, 6:36 PM

Change 637752 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/mediawiki-config@master] Reduce size of frwiki featuredfeeds to default

https://gerrit.wikimedia.org/r/637752

Change 637752 merged by jenkins-bot:
[operations/mediawiki-config@master] Reduce size of frwiki featuredfeeds to default

https://gerrit.wikimedia.org/r/637752

Mentioned in SAL (#wikimedia-operations) [2020-10-30T18:47:58Z] <cdanis> ✔️ cdanis@deploy1001.eqiad.wmnet /srv/mediawiki-staging 🕝☕ scap sync-file wmf-config/InitialiseSettings.php 'lower frwiki featured feeds limit 1a41ef634 T266865'

Mentioned in SAL (#wikimedia-operations) [2020-10-30T18:48:06Z] <cdanis@deploy1001> Synchronized wmf-config/InitialiseSettings.php: lower frwiki featured feeds limit 1a41ef634 T266865 (duration: 05m 14s)

CDanis closed this task as Resolved.EditedFri, Oct 30, 6:55 PM
CDanis claimed this task.
CDanis added subscribers: Urbanecm, Legoktm.

Approx 23:00 on 28 Oct, the size of the featured feed for frwiki started to become too large to be stored as a value in our memcached.
Memcached error for key "WANCache:frwiki:featured-feeds:1:fr|#|v" on server "127.0.0.1:11213": ITEM TOO BIG


https://logstash.wikimedia.org/goto/e0d82ea5e0c9883a61b200eff570a140

This means the feed contents were being recomputed on each main page render, even though (as pointed out by @Legoktm) the main page doesn't actually need the contents, only some metadata. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/FeaturedFeeds/+/refs/heads/master/includes/FeaturedFeeds.php#116
(Followup AI: fix this :)

Long ago, frwiki's default feed length (in days) was set to 60, well above the default of 10.
We reverted the limit to the default.

Special thanks to @Legoktm and @Urbanecm for helping track this down!

Long ago, frwiki's default feed length (in days) was set to 60, well above the default of 10.
We reverted the limit to the default.

I'm not sure what the historical reasoning behind this was, but as long as people open their feed readers at least once every 10 days, they shouldn't miss any content, so I would say the practical impact of this change will be pretty low.

greg added a subscriber: greg.Sat, Oct 31, 4:03 AM

Long ago, frwiki's default feed length (in days) was set to 60, well above the default of 10.
We reverted the limit to the default.

I'm not sure what the historical reasoning behind this was

https://phabricator.wikimedia.org/T167617#3349566

Dzahn added a subscriber: Dzahn.Sat, Oct 31, 5:21 PM

I would say the practical impact of this change will be pretty low.

Yes, the impact should only be "users reading the aggregated feed on fr.planet.wikimedia.org get fewer items included in that feed".

The limit of 60 _items per daily feed_ was reduced to the default of 10.

Left comments about it at https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Le_Bistro#Temps_de_chargement_de_la_page_d'accueil

Lam1982019 set Due Date to Sat, Oct 31, 5:00 PM.Sun, Nov 1, 1:09 PM
Aklapper removed Due Date.Sun, Nov 1, 1:46 PM

+Wikimedia-Incident if this was considered an incident