Page MenuHomePhabricator
Create Task
Maniphest T93141

Convert various core/extension cache users to the WAN cache as needed
Closed, ResolvedPublic
Actions

Description

Per http://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki#Design_implications

This could even be part of some hackathon type sprinting :)

Patches could not be merged until the new cache was up and configure of course.

Details

SubjectRepoBranchLines +/-
Conversion to WANObjectCachemediawiki/extensions/CentralAuthmaster+8 -9
Conversion to using WAN cachemediawiki/extensions/Gadgetswmf/1.26wmf7+91 -54
Conversion to using WAN cachemediawiki/extensions/Gadgetsmaster+91 -54
Converted FileBackendStore to using the WAN cachemediawiki/coremaster+24 -21
Converted User object cache to the WAN cachemediawiki/coremaster+10 -11
Conversion to using WAN cachemediawiki/extensions/AbuseFiltermaster+30 -44
Conversion to using WAN cachemediawiki/extensions/FeaturedFeedsmaster+15 -12
Converted User touched key to using the WAN cachemediawiki/coremaster+8 -9
Converted LocalFile to using the WAN cachemediawiki/coremaster+25 -17
Converted ChangeTags to using the WAN cachemediawiki/coremaster+29 -22
Converted Interwiki using WAN cachemediawiki/coremaster+17 -4
Converted sidebar cache to the WAN cachemediawiki/coremaster+6 -5
Converted LocalRepo to using the WAN cachemediawiki/coremaster+8 -7
Use the WAN cache for cachingmediawiki/extensions/WikiGrokmaster+8 -6
Converted WikiSet to using the WAN cachemediawiki/extensions/CentralAuthmaster+9 -9
Conversion to using WAN cachemediawiki/extensions/Petitionmaster+19 -26
Conversion to using WAN cachemediawiki/extensions/PageTriagemaster+11 -56
Converted InfoAction to using the WAN cachemediawiki/coremaster+6 -5
Conversion to using WAN cachemediawiki/extensions/UploadWizardmaster+10 -13
Conversion to using WAN cachemediawiki/extensions/SpamBlacklistmaster+10 -7
Conversion to using WAN cachemediawiki/extensions/TitleBlacklistmaster+12 -8
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aaron added projects: Epic, MediaWiki-Core-Team.Mar 18 2015, 9:32 PM
Liuxinyu970226 subscribed.Mar 21 2015, 4:25 AM
• Gilles subscribed.Apr 2 2015, 12:24 PM
• Gilles edited projects, added Sustainability; removed MediaWiki-Core-Team.Apr 7 2015, 7:30 AM
gerritbot added a comment.Apr 30 2015, 1:29 AM

Change 206975 had a related patch set uploaded (by Aaron Schulz):
Converted sidebar cache to the WAN cache

https://gerrit.wikimedia.org/r/206975

gerritbot added a project: Patch-For-Review.Apr 30 2015, 1:29 AM

Change 206968 had a related patch set uploaded (by Aaron Schulz):
Converted LocalRepo to using the WAN cache

https://gerrit.wikimedia.org/r/206968

gerritbot added a comment.Apr 30 2015, 1:29 AM

Change 206989 had a related patch set uploaded (by Aaron Schulz):
Converted InfoAction to using the WAN cache

https://gerrit.wikimedia.org/r/206989

gerritbot added a comment.Apr 30 2015, 1:30 AM

Change 206998 had a related patch set uploaded (by Aaron Schulz):
Converted User touched key to using the WAN cache

https://gerrit.wikimedia.org/r/206998

gerritbot added a comment.Apr 30 2015, 1:30 AM

Change 207001 had a related patch set uploaded (by Aaron Schulz):
Converted User object cache to the WAN cache

https://gerrit.wikimedia.org/r/207001

gerritbot added a comment.Apr 30 2015, 1:30 AM

Change 207004 had a related patch set uploaded (by Aaron Schulz):
Converted LocalFile to using the WAN cache

https://gerrit.wikimedia.org/r/207004

gerritbot added a comment.Apr 30 2015, 1:30 AM

Change 206966 had a related patch set uploaded (by Aaron Schulz):
Converted ChangeTags to using the WAN cache

https://gerrit.wikimedia.org/r/206966

gerritbot added a comment.Apr 30 2015, 1:30 AM

Change 206958 had a related patch set uploaded (by Aaron Schulz):
Converted FileBackendStore to using the WAN cache

https://gerrit.wikimedia.org/r/206958

gerritbot added a comment.Apr 30 2015, 1:31 AM

Change 207638 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207638

gerritbot added a comment.Apr 30 2015, 1:32 AM

Change 207639 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207639

gerritbot added a comment.Apr 30 2015, 1:33 AM

Change 207646 had a related patch set uploaded (by Aaron Schulz):
Converted Interwiki using WAN cache

https://gerrit.wikimedia.org/r/207646

gerritbot added a comment.Apr 30 2015, 1:33 AM

Change 207652 had a related patch set uploaded (by Aaron Schulz):
Converted WikiSet to using the WAN cache

https://gerrit.wikimedia.org/r/207652

gerritbot added a comment.Apr 30 2015, 1:35 AM

Change 207630 had a related patch set uploaded (by Aaron Schulz):
Use the WAN cache for caching

https://gerrit.wikimedia.org/r/207630

aaron mentioned this in T91815: Audit cache usage to see what can be DC-local and what needs cross-DC purges.Apr 30 2015, 1:36 AM
aaron updated the task description. (Show Details)
aaron set Security to None.
aaron moved this task from Tag to Doing on the Sustainability board.Apr 30 2015, 5:24 AM
gerritbot added a comment.Apr 30 2015, 5:24 AM

Change 207729 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207729

gerritbot added a comment.Apr 30 2015, 5:32 AM

Change 207730 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207730

gerritbot added a comment.Apr 30 2015, 6:59 PM

Change 207891 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207891

gerritbot added a comment.Apr 30 2015, 7:18 PM

Change 207897 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207897

gerritbot added a comment.Apr 30 2015, 8:04 PM

Change 207915 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207915

gerritbot added a comment.May 5 2015, 10:40 AM

Change 207638 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207638

Diffusion mentioned this in rMEXTf076e2d994ff: Updated mediawiki/extensions Project: mediawiki/extensions/TitleBlacklist….May 5 2015, 10:40 AM
aaron mentioned this in rETBL72cff297541d: Conversion to using WAN cache.May 5 2015, 10:44 AM
gerritbot added a comment.May 5 2015, 10:46 AM

Change 207639 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207639

Diffusion mentioned this in rMEXT5c45bd5f6378: Updated mediawiki/extensions Project: mediawiki/extensions/SpamBlacklist….May 5 2015, 10:47 AM
gerritbot added a comment.May 5 2015, 10:47 AM

Change 207897 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207897

Diffusion mentioned this in rMEXTf2ee5cede849: Updated mediawiki/extensions Project: mediawiki/extensions/UploadWizard….May 5 2015, 10:47 AM
aaron mentioned this in rEUWIece27d0c923f: Conversion to using WAN cache.May 5 2015, 10:48 AM
aaron mentioned this in rESPBca55c42a1e88: Conversion to using WAN cache.May 5 2015, 11:12 AM
gerritbot added a comment.May 5 2015, 11:41 AM

Change 206989 merged by jenkins-bot:
Converted InfoAction to using the WAN cache

https://gerrit.wikimedia.org/r/206989

aaron mentioned this in rMW19878798efa0: Converted InfoAction to using the WAN cache.May 5 2015, 11:41 AM
gerritbot added a comment.May 5 2015, 1:31 PM

Change 207915 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207915

Diffusion mentioned this in rMEXT0971f11d0fdb: Updated mediawiki/extensions Project: mediawiki/extensions/PageTriage….May 5 2015, 1:32 PM
aaron mentioned this in rEPTRa5f0ed2b161c: Conversion to using WAN cache.May 5 2015, 1:47 PM
gerritbot added a comment.May 5 2015, 1:58 PM

Change 207891 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207891

aaron mentioned this in rMEXT6262cb035269: Updated mediawiki/extensions Project: mediawiki/extensions/Petition….May 5 2015, 1:59 PM
aaron mentioned this in rEPETba0026f69282: Conversion to using WAN cache.May 5 2015, 2:13 PM
gerritbot added a comment.May 7 2015, 7:30 PM

Change 206966 merged by jenkins-bot:
Converted ChangeTags to using the WAN cache

https://gerrit.wikimedia.org/r/206966

gerritbot added a comment.May 7 2015, 7:30 PM

Change 207652 merged by jenkins-bot:
Converted WikiSet to using the WAN cache

https://gerrit.wikimedia.org/r/207652

aaron mentioned this in rMW134703b0a075: Converted ChangeTags to using the WAN cache.May 7 2015, 7:30 PM
aaron mentioned this in rECAU93b15188f0c5: Converted WikiSet to using the WAN cache.
Diffusion mentioned this in rMEXT37e34c18ae36: Updated mediawiki/extensions Project: mediawiki/extensions/CentralAuth….May 7 2015, 7:30 PM
gerritbot added a comment.May 7 2015, 7:39 PM

Change 206968 merged by jenkins-bot:
Converted LocalRepo to using the WAN cache

https://gerrit.wikimedia.org/r/206968

aaron mentioned this in rMW2f23a508a2d5: Converted LocalRepo to using the WAN cache.May 7 2015, 7:40 PM
gerritbot added a comment.May 7 2015, 7:43 PM

Change 207630 merged by jenkins-bot:
Use the WAN cache for caching

https://gerrit.wikimedia.org/r/207630

aaron mentioned this in rEWGRe1b6408549b4: Use the WAN cache for caching.May 7 2015, 7:43 PM
Diffusion mentioned this in rMEXT6d31efcaa44f: Updated mediawiki/extensions Project: mediawiki/extensions/WikiGrok….May 7 2015, 7:43 PM
gerritbot added a comment.May 8 2015, 8:06 AM

Change 206975 merged by jenkins-bot:
Converted sidebar cache to the WAN cache

https://gerrit.wikimedia.org/r/206975

aaron mentioned this in rMWe9a609c7afbd: Converted sidebar cache to the WAN cache.May 8 2015, 8:07 AM
gerritbot added a comment.May 8 2015, 8:29 AM

Change 207646 merged by jenkins-bot:
Converted Interwiki using WAN cache

https://gerrit.wikimedia.org/r/207646

aaron mentioned this in rMW4a74dbf1988d: Converted Interwiki using WAN cache.May 8 2015, 8:29 AM
Cenarium subscribed.May 9 2015, 12:17 AM

Please see comment in commit 206966.

gerritbot added a comment.May 11 2015, 4:21 PM

Change 207004 merged by jenkins-bot:
Converted LocalFile to using the WAN cache

https://gerrit.wikimedia.org/r/207004

aaron mentioned this in rMW429a22cd088d: Converted LocalFile to using the WAN cache.May 11 2015, 4:21 PM
gerritbot added a comment.May 11 2015, 4:52 PM

Change 206998 merged by jenkins-bot:
Converted User touched key to using the WAN cache

https://gerrit.wikimedia.org/r/206998

aaron mentioned this in rMW4a8da410795d: Converted User touched key to using the WAN cache.May 11 2015, 4:52 PM
gerritbot added a comment.May 12 2015, 3:05 PM

Change 207729 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207729

Diffusion mentioned this in rMEXTaeacaca3530e: Updated mediawiki/extensions Project: mediawiki/extensions/FeaturedFeeds….May 12 2015, 3:06 PM
aaron mentioned this in rEFFDc2f3577003da: Conversion to using WAN cache.May 12 2015, 3:34 PM
gerritbot added a comment.May 13 2015, 5:54 AM

Change 210647 had a related patch set uploaded (by Aaron Schulz):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/210647

gerritbot added a comment.May 13 2015, 10:46 PM

Change 210647 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/210647

Diffusion mentioned this in rMEXTcffb8d0b7134: Updated mediawiki/extensions Project: mediawiki/extensions/AbuseFilter….May 13 2015, 10:47 PM
aaron mentioned this in rEABF60d1eebb6b8a: Conversion to using WAN cache.May 13 2015, 10:47 PM
aaron moved this task from Doing to Blocked on the Sustainability board.May 15 2015, 2:58 AM
gerritbot added a comment.May 18 2015, 6:38 AM

Change 207001 merged by jenkins-bot:
Converted User object cache to the WAN cache

https://gerrit.wikimedia.org/r/207001

aaron mentioned this in rMWb614290f73a4: Converted User object cache to the WAN cache.May 18 2015, 6:38 AM
gerritbot added a comment.May 18 2015, 8:34 AM

Change 206958 merged by jenkins-bot:
Converted FileBackendStore to using the WAN cache

https://gerrit.wikimedia.org/r/206958

aaron mentioned this in rMW48cffdff8219: Converted FileBackendStore to using the WAN cache.May 18 2015, 8:34 AM
gerritbot added a comment.May 23 2015, 11:04 PM

Change 207730 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/207730

gerritbot added a comment.May 23 2015, 11:04 PM

Change 213186 had a related patch set uploaded (by Ori.livneh):
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/213186

aaron mentioned this in rMEXTa780ec87b9dc: Updated mediawiki/extensions Project: mediawiki/extensions/Gadgets….May 23 2015, 11:04 PM
matmarex added subscribers: matmarex, Ankry.May 23 2015, 11:04 PM
gerritbot added a comment.May 23 2015, 11:05 PM

Change 213186 merged by jenkins-bot:
Conversion to using WAN cache

https://gerrit.wikimedia.org/r/213186

matmarex removed subscribers: Ankry, matmarex.May 23 2015, 11:05 PM
ori mentioned this in rEGAD9add4fcca165: Conversion to using WAN cache.May 23 2015, 11:22 PM
ori mentioned this in rEGAD11e1c51d6b34: Conversion to using WAN cache.
Nemo_bis subscribed.May 24 2015, 6:11 AM
In T93141#1273954, @Cenarium wrote:

Please see comment in commit 206966.

Since the delete operation of WANObjectCache prevents any caching for ten seconds, this had the effect of preventing any caching of tag usage statistics whenever a tag is added (or removed) for the next ten seconds.
On highly active wikis like wikipedias, where tags are added every few seconds, this means that caching would almost never occur. In all likelihood, this would make Special:Tags unusable.

https://en.wikipedia.org/wiki/Special:Tags WFM, is all well?

aaron added a comment.May 24 2015, 7:44 AM
In T93141#1306066, @Nemo_bis wrote:
In T93141#1273954, @Cenarium wrote:

Please see comment in commit 206966.

Since the delete operation of WANObjectCache prevents any caching for ten seconds, this had the effect of preventing any caching of tag usage statistics whenever a tag is added (or removed) for the next ten seconds.
On highly active wikis like wikipedias, where tags are added every few seconds, this means that caching would almost never occur. In all likelihood, this would make Special:Tags unusable.

https://en.wikipedia.org/wiki/Special:Tags WFM, is all well?

A follow-up patch was made in 3782a0d53bbe to improve performance post-purges to avoid stampedes.

The query is still awful and the constant purges seem kind of stupid (it probably can just purge when the new count is 0 or 1). Also I wonder if it really needs to count 2mil items or if it can clip the count. Also self::tagUsageStatistics() could use a process cache since permission checks call it several times for each tag. Lots of room for improvement.

Cenarium added a comment.May 25 2015, 2:33 AM
In T93141#1306129, @aaron wrote:

The query is still awful and the constant purges seem kind of stupid (it probably can just purge when the new count is 0 or 1). Also I wonder if it really needs to count 2mil items or if it can clip the count. Also self::tagUsageStatistics() could use a process cache since permission checks call it several times for each tag. Lots of room for improvement.

I've been trying to address these and other issues in commit 201905.

aaron closed this task as Resolved.Jul 2 2015, 7:27 PM
Liuxinyu970226 unsubscribed.Jul 3 2015, 1:27 AM
aaron mentioned this in rMREL60d1eebb6b8a: Conversion to using WAN cache.Jul 14 2015, 6:11 PM
gerritbot added a comment.Jul 16 2015, 6:53 AM

Change 225013 had a related patch set uploaded (by Aaron Schulz):
Conversion to WANObjectCache

https://gerrit.wikimedia.org/r/225013

gerritbot added a comment.Aug 4 2015, 1:42 AM

Change 225013 merged by jenkins-bot:
Conversion to WANObjectCache

https://gerrit.wikimedia.org/r/225013

Diffusion mentioned this in rMEXT37cd643209b1: Updated mediawiki/extensions Project: mediawiki/extensions/CentralAuth….Aug 4 2015, 1:42 AM
aaron mentioned this in rECAUde339ea33115: Conversion to WANObjectCache.Aug 4 2015, 1:43 AM
• Forrestbot added a project: WMF-deploy-2015-08-04_(1.26wmf17).Aug 4 2015, 2:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL