Page MenuHomePhabricator

Reduce parser cache retention temporarily for DiscussionTools
Closed, ResolvedPublic

Description

This task represents the work with reducing how long the parser cache retains data.

Requirements

  1. The Data-Persistence team, in conjunction with the Editing-team and Performance-Team, defines how the often the parser cache should be refreshed/cleared
  2. The Data-Persistence team writes and deploys the patch(es) required to put "1." into effect.

Details and strategy

The parser cache databases have had their size increase for several months in a way that has started to exceed the operational thresholds set by the DBAs. This is believed to be (at least in part) due to the DiscussionTools roll-out. Refer to T280599 for more context on that.

We have in the past decade sometimes exceeded these thresholds from organic growth, in which case, if new capacity/hardware did not arrive in time, we would temporary sacrifice page view performance by reducing the retention time of the ParserCache. This is something we'd like to avoid, and if not possible, to do for as little time as possible.

The alternative plan by @Krinkle was to take this oppertunity to see if we can adopt a pattern of more granularly controlling and localising these costs instead of having only a single lever. As such, the plan is to reduce the parser cache expiry of talk page specifically.

Cost. As with any change in retention, such configuration change only affects how new entries are stored. It does not work retroactively and would take at least a month of churn before we see whether it worked. In order to resolve the current operational risk we must do something sooner, namely to retroactively evict blobs older than a certain date as well. This retroactive action requires 3-4 days of labourious work to do in such a way that the database servers remain responsive to live traffic. During these 3-4 days of work, the spare parser cache servers will employed, which have as side-effect that they will be partially unable to serve data more than a day old. This has direct impact on page view performance.

Unknowns: We don't know the distribution of talk pages in the parser cache. We don't know what percentage of talk pages gets frequently modified and rotated within a few days vs stays the same and benefits from caching for longer. For example, if the vast majority of talk page views are within a week of them being modified (and thus added to the cache) then caching them for longer does not benefit much, and shorting their retention would gain us space. On the other hand, if people regularly view stale discussion pages without then changing them, that means shortening the expiry would just lead to the same stale pages being re-cached as new entries and not significantly reducing the used space.

Risk: There is a chance that the new DiscussionTools would (still) take up more space than the amount of space we can reclaim by reducing its retention. As such, if all we did is reduce the DT cache expiry, there is a chance we would have to do the 3-4 days of reclaim work a second time, and also pay the end-user penalty again.

Strategy: We will avoid this risk by performing the DT mitigation and our fallback mitigation at the same time. The fallback mitigation is to reduce the cache expiry of Wikipedia articles. This means we will definitely reclaim the amount of space we need. Then, after we have found our new equilibrium we determine how far we can ramp the Wikipedia article cache expiry back up. (Either in full if the DT strategy worked out as hoped, or only in part if it didn't).

Done

30 days after this is resolved, the follow-up task T280604 will kick-in, which covers ramping the overall expiry back up.

Event Timeline

Krinkle renamed this task from Deploy config change to reduce parser cache data retention time to Reduce parser cache retention temporarily for DiscussionTools.May 4 2021, 4:44 PM
Krinkle claimed this task.
Krinkle triaged this task as High priority.
Krinkle edited projects, added Performance-Team; removed Performance-Team (Radar).
Krinkle updated the task description. (Show Details)
Krinkle moved this task from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Krinkle added subscribers: Marostegui, Krinkle.

Change 685164 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] WikiPage: Document triggerOpportunisticLinksUpdate and related code

https://gerrit.wikimedia.org/r/685164

Change 685180 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/DiscussionTools@master] Allow talk pages to have a different ParserCache expiry

https://gerrit.wikimedia.org/r/685180

Change 685181 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] Temporarily shorten $wgParserCacheExpireTime from 30 to 22 days

https://gerrit.wikimedia.org/r/685181

Marostegui moved this task from Triage to In progress on the DBA board.

Change 685222 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] parsercachepurging.pp: Reduce parsercache retention to 21 days

https://gerrit.wikimedia.org/r/685222

Change 685164 merged by jenkins-bot:

[mediawiki/core@master] WikiPage: Document triggerOpportunisticLinksUpdate and related code

https://gerrit.wikimedia.org/r/685164

@Krinkle this is ready to go whenever you are done with the manual script run: https://gerrit.wikimedia.org/r/685222

Change 685222 merged by Marostegui:

[operations/puppet@production] parsercachepurging.pp: Reduce parsercache retention to 21 days

https://gerrit.wikimedia.org/r/685222

Merged the script. Tonight the script will purge everything older than 21 days.

Change 685878 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] purgeParserCache.php: Remove carriage return printer, minor cleanup

https://gerrit.wikimedia.org/r/685878

There was a script running from May 2nd still on mwmaint1002, which was stuck. I have killed it and as soon as I did that, the new one (the one purging stuff older than 21 days) spawned, so it is now running.

There was a script running from May 2nd still on mwmaint1002, which was stuck. I have killed it […]

I noticed it was running for 5 days indeed, but tailing its journal/syslog showed that it was making progress (about 1% every 4 hours).

The new instance of the maintenance job seems to be similarly slow, having done 2.4% in the 13 hours so far.

krinkle at mwmaint1002.eqiad.wmnet in /var/log/mediawiki/mediawiki_job_parser_cache_purging/
$ fgrep "Deleting" syslog.log | tail -1
May  7 05:23:37 mwmaint1002 mediawiki_job_parser_cache_purging: Deleting objects expiring before 05:23, 16 May 2021
$ tail -n2 syslog.log
…
May  7 14:58:45 mwmaint1002 mediawiki_job_parser_cache_purging 1.72%
May  7 16:59:58 mwmaint1002 mediawiki_job_parser_cache_purging 2.07%
May  7 18:51:12 mwmaint1002 mediawiki_job_parser_cache_purging 2.42%

And ithe runtime has been gradually getting worse for a while:

1# [19:15 UTC] krinkle at mwmaint1002.eqiad.wmnet in /var/log/mediawiki/mediawiki_job_parser_cache_purging
2# $ fgrep "Deleting" syslog.log
3Apr 10 01:00:03 mwmaint1002 mediawiki_job_parser_cache_purging[146571]: Deleting objects expiring before 01:00, 10 April 2020
4Apr 11 03:56:45 mwmaint1002 mediawiki_job_parser_cache_purging[240365]: Deleting objects expiring before 03:56, 11 April 2020
5Apr 12 08:12:32 mwmaint1002 mediawiki_job_parser_cache_purging[72089]: Deleting objects expiring before 08:12, 12 April 2020
6Apr 13 07:29:56 mwmaint1002 mediawiki_job_parser_cache_purging[2846]: Deleting objects expiring before 07:29, 13 April 2020
7Apr 14 02:38:09 mwmaint1002 mediawiki_job_parser_cache_purging[210524]: Deleting objects expiring before 02:38, 14 April 2020
8Apr 15 01:00:03 mwmaint1002 mediawiki_job_parser_cache_purging[83431]: Deleting objects expiring before 01:00, 15 April 2020
9Apr 16 01:00:06 mwmaint1002 mediawiki_job_parser_cache_purging[66915]: Deleting objects expiring before 01:00, 16 April 2020
10Apr 17 01:15:19 mwmaint1002 mediawiki_job_parser_cache_purging[12688]: Deleting objects expiring before 01:15, 17 April 2020
11Apr 18 01:00:01 mwmaint1002 mediawiki_job_parser_cache_purging[204947]: Deleting objects expiring before 01:00, 18 April 2020
12Apr 19 13:07:02 mwmaint1002 mediawiki_job_parser_cache_purging[182094]: Deleting objects expiring before 13:07, 19 April 2020
13Apr 21 20:59:35 mwmaint1002 mediawiki_job_parser_cache_purging[250916]: Deleting objects expiring before 20:59, 21 April 2020
14Apr 24 19:47:46 mwmaint1002 mediawiki_job_parser_cache_purging[101035]: Deleting objects expiring before 19:47, 24 April 2020
15Apr 27 23:09:18 mwmaint1002 mediawiki_job_parser_cache_purging[206784]: Deleting objects expiring before 23:09, 27 April 2020
16Apr 30 16:53:08 mwmaint1002 mediawiki_job_parser_cache_purging[205448]: Deleting objects expiring before 16:53, 30 April 2020
17May 3 00:37:27 mwmaint1002 mediawiki_job_parser_cache_purging[42048]: Deleting objects expiring before 00:37, 3 May 2020
18May 5 05:13:29 mwmaint1002 mediawiki_job_parser_cache_purging[20388]: Deleting objects expiring before 05:13, 5 May 2020
19May 7 16:30:25 mwmaint1002 mediawiki_job_parser_cache_purging[19386]: Deleting objects expiring before 16:30, 7 May 2020
20May 9 21:54:40 mwmaint1002 mediawiki_job_parser_cache_purging[259760]: Deleting objects expiring before 21:54, 9 May 2020
21May 12 03:47:45 mwmaint1002 mediawiki_job_parser_cache_purging[13882]: Deleting objects expiring before 03:47, 12 May 2020
22May 14 04:12:29 mwmaint1002 mediawiki_job_parser_cache_purging[164959]: Deleting objects expiring before 04:12, 14 May 2020
23May 16 02:39:31 mwmaint1002 mediawiki_job_parser_cache_purging[32112]: Deleting objects expiring before 02:39, 16 May 2020
24May 18 02:41:09 mwmaint1002 mediawiki_job_parser_cache_purging[196859]: Deleting objects expiring before 02:41, 18 May 2020
25May 20 07:12:25 mwmaint1002 mediawiki_job_parser_cache_purging[167348]: Deleting objects expiring before 07:12, 20 May 2020
26May 22 13:35:47 mwmaint1002 mediawiki_job_parser_cache_purging[186721]: Deleting objects expiring before 13:35, 22 May 2020
27May 25 06:03:37 mwmaint1002 mediawiki_job_parser_cache_purging[149790]: Deleting objects expiring before 06:03, 25 May 2020
28May 28 13:49:01 mwmaint1002 mediawiki_job_parser_cache_purging[190184]: Deleting objects expiring before 13:49, 28 May 2020
29Jun 1 05:22:29 mwmaint1002 mediawiki_job_parser_cache_purging[41516]: Deleting objects expiring before 05:22, 1 June 2020
30Jun 3 14:01:01 mwmaint1002 mediawiki_job_parser_cache_purging[161911]: Deleting objects expiring before 14:01, 3 June 2020
31Jun 5 23:24:21 mwmaint1002 mediawiki_job_parser_cache_purging[19792]: Deleting objects expiring before 23:24, 5 June 2020
32Jun 9 18:49:46 mwmaint1002 mediawiki_job_parser_cache_purging[228933]: Deleting objects expiring before 18:49, 9 June 2020
33Jun 13 06:37:17 mwmaint1002 mediawiki_job_parser_cache_purging[33396]: Deleting objects expiring before 06:37, 13 June 2020
34Jun 16 06:29:11 mwmaint1002 mediawiki_job_parser_cache_purging[84032]: Deleting objects expiring before 06:29, 16 June 2020
35Jun 19 06:14:53 mwmaint1002 mediawiki_job_parser_cache_purging[92680]: Deleting objects expiring before 06:14, 19 June 2020
36Jun 22 17:19:58 mwmaint1002 mediawiki_job_parser_cache_purging[88936]: Deleting objects expiring before 17:19, 22 June 2020
37Jun 26 11:15:00 mwmaint1002 mediawiki_job_parser_cache_purging[156464]: Deleting objects expiring before 11:15, 26 June 2020
38Jun 30 09:46:00 mwmaint1002 mediawiki_job_parser_cache_purging[205484]: Deleting objects expiring before 09:46, 30 June 2020
39Jul 3 17:42:57 mwmaint1002 mediawiki_job_parser_cache_purging[221089]: Deleting objects expiring before 17:42, 3 July 2020
40Jul 6 21:08:00 mwmaint1002 mediawiki_job_parser_cache_purging[119347]: Deleting objects expiring before 21:08, 6 July 2020
41Jul 9 17:05:20 mwmaint1002 mediawiki_job_parser_cache_purging[36442]: Deleting objects expiring before 17:05, 9 July 2020
42Jul 12 08:39:17 mwmaint1002 mediawiki_job_parser_cache_purging[128375]: Deleting objects expiring before 08:39, 12 July 2020
43Jul 14 22:00:43 mwmaint1002 mediawiki_job_parser_cache_purging[100580]: Deleting objects expiring before 22:00, 14 July 2020
44Jul 17 08:42:25 mwmaint1002 mediawiki_job_parser_cache_purging[24302]: Deleting objects expiring before 08:42, 17 July 2020
45Jul 20 01:20:05 mwmaint1002 mediawiki_job_parser_cache_purging[64583]: Deleting objects expiring before 01:20, 20 July 2020
46Jul 22 23:32:32 mwmaint1002 mediawiki_job_parser_cache_purging[253940]: Deleting objects expiring before 23:32, 22 July 2020
47Jul 26 00:31:13 mwmaint1002 mediawiki_job_parser_cache_purging[40242]: Deleting objects expiring before 00:31, 26 July 2020
48Jul 29 06:18:16 mwmaint1002 mediawiki_job_parser_cache_purging[191576]: Deleting objects expiring before 06:18, 29 July 2020
49Aug 1 08:44:23 mwmaint1002 mediawiki_job_parser_cache_purging[29297]: Deleting objects expiring before 08:44, 1 August 2020
50Aug 4 10:56:00 mwmaint1002 mediawiki_job_parser_cache_purging[173135]: Deleting objects expiring before 10:56, 4 August 2020
51Aug 7 22:14:41 mwmaint1002 mediawiki_job_parser_cache_purging[39261]: Deleting objects expiring before 22:14, 7 August 2020
52Aug 11 06:58:47 mwmaint1002 mediawiki_job_parser_cache_purging[258229]: Deleting objects expiring before 06:58, 11 August 2020
53Aug 14 07:15:29 mwmaint1002 mediawiki_job_parser_cache_purging[54964]: Deleting objects expiring before 07:15, 14 August 2020
54Aug 17 06:40:56 mwmaint1002 mediawiki_job_parser_cache_purging[100453]: Deleting objects expiring before 06:40, 17 August 2020
55Aug 20 01:53:44 mwmaint1002 mediawiki_job_parser_cache_purging[231161]: Deleting objects expiring before 01:53, 20 August 2020
56Aug 22 22:47:23 mwmaint1002 mediawiki_job_parser_cache_purging[185114]: Deleting objects expiring before 22:47, 22 August 2020
57Aug 25 20:14:44 mwmaint1002 mediawiki_job_parser_cache_purging[175794]: Deleting objects expiring before 20:14, 25 August 2020
58Aug 28 21:03:15 mwmaint1002 mediawiki_job_parser_cache_purging[38037]: Deleting objects expiring before 21:03, 28 August 2020
59Sep 1 04:16:18 mwmaint1002 mediawiki_job_parser_cache_purging[254032]: Deleting objects expiring before 04:16, 1 September 2020
60Oct 28 01:00:07 mwmaint1002 mediawiki_job_parser_cache_purging[85535]: Deleting objects expiring before 01:00, 28 October 2020
61Nov 1 13:27:46 mwmaint1002 mediawiki_job_parser_cache_purging[111100]: Deleting objects expiring before 13:27, 1 November 2020
62Nov 5 03:02:09 mwmaint1002 mediawiki_job_parser_cache_purging[125718]: Deleting objects expiring before 03:02, 5 November 2020
63Nov 8 06:52:23 mwmaint1002 mediawiki_job_parser_cache_purging[210463]: Deleting objects expiring before 06:52, 8 November 2020
64Nov 11 04:04:49 mwmaint1002 mediawiki_job_parser_cache_purging[46811]: Deleting objects expiring before 04:04, 11 November 2020
65Nov 13 18:05:08 mwmaint1002 mediawiki_job_parser_cache_purging[23469]: Deleting objects expiring before 18:05, 13 November 2020
66Nov 16 08:24:39 mwmaint1002 mediawiki_job_parser_cache_purging[25291]: Deleting objects expiring before 08:24, 16 November 2020
67Nov 18 11:56:21 mwmaint1002 mediawiki_job_parser_cache_purging[130565]: Deleting objects expiring before 11:56, 18 November 2020
68Nov 21 18:06:30 mwmaint1002 mediawiki_job_parser_cache_purging[145145]: Deleting objects expiring before 18:06, 21 November 2020
69Nov 27 06:01:34 mwmaint1002 mediawiki_job_parser_cache_purging[149815]: Deleting objects expiring before 06:01, 27 November 2020
70Dec 3 06:56:15 mwmaint1002 mediawiki_job_parser_cache_purging[253031]: Deleting objects expiring before 06:56, 3 December 2020
71Dec 9 03:39:20 mwmaint1002 mediawiki_job_parser_cache_purging[144320]: Deleting objects expiring before 03:39, 9 December 2020
72Dec 14 05:59:59 mwmaint1002 mediawiki_job_parser_cache_purging[111951]: Deleting objects expiring before 05:59, 14 December 2020
73Dec 18 08:23:34 mwmaint1002 mediawiki_job_parser_cache_purging[49615]: Deleting objects expiring before 08:23, 18 December 2020
74Dec 21 09:08:16 mwmaint1002 mediawiki_job_parser_cache_purging[149733]: Deleting objects expiring before 09:08, 21 December 2020
75Dec 24 17:45:30 mwmaint1002 mediawiki_job_parser_cache_purging[13185]: Deleting objects expiring before 17:45, 24 December 2020
76Dec 27 23:09:22 mwmaint1002 mediawiki_job_parser_cache_purging[236394]: Deleting objects expiring before 23:09, 27 December 2020
77Dec 31 04:37:15 mwmaint1002 mediawiki_job_parser_cache_purging[209639]: Deleting objects expiring before 04:37, 31 December 2020
78Jan 3 10:12:26 mwmaint1002 mediawiki_job_parser_cache_purging[9945]: Deleting objects expiring before 10:12, 3 January 2021
79Jan 6 22:31:39 mwmaint1002 mediawiki_job_parser_cache_purging[145468]: Deleting objects expiring before 22:31, 6 January 2021
80Jan 10 11:40:57 mwmaint1002 mediawiki_job_parser_cache_purging[37493]: Deleting objects expiring before 11:40, 10 January 2021
81Jan 14 08:21:18 mwmaint1002 mediawiki_job_parser_cache_purging[104818]: Deleting objects expiring before 08:21, 14 January 2021
82Jan 18 10:04:29 mwmaint1002 mediawiki_job_parser_cache_purging[201674]: Deleting objects expiring before 10:04, 18 January 2021
83Jan 23 02:11:11 mwmaint1002 mediawiki_job_parser_cache_purging[131431]: Deleting objects expiring before 02:11, 23 January 2021
84Jan 28 03:31:21 mwmaint1002 mediawiki_job_parser_cache_purging[17858]: Deleting objects expiring before 03:31, 28 January 2021
85Feb 2 01:50:58 mwmaint1002 mediawiki_job_parser_cache_purging[155799]: Deleting objects expiring before 01:50, 2 February 2021
86Feb 7 02:45:33 mwmaint1002 mediawiki_job_parser_cache_purging[99534]: Deleting objects expiring before 02:45, 7 February 2021
87Feb 12 01:00:02 mwmaint1002 mediawiki_job_parser_cache_purging[54460]: Deleting objects expiring before 01:00, 12 February 2021
88Feb 17 08:06:44 mwmaint1002 mediawiki_job_parser_cache_purging[109741]: Deleting objects expiring before 08:06, 17 February 2021
89Feb 22 07:49:48 mwmaint1002 mediawiki_job_parser_cache_purging[213157]: Deleting objects expiring before 07:49, 22 February 2021
90Feb 27 15:09:30 mwmaint1002 mediawiki_job_parser_cache_purging[334]: Deleting objects expiring before 15:09, 27 February 2021
91Mar 5 00:05:42 mwmaint1002 mediawiki_job_parser_cache_purging[7982]: Deleting objects expiring before 00:05, 5 March 2021
92Mar 10 15:46:03 mwmaint1002 mediawiki_job_parser_cache_purging[66538]: Deleting objects expiring before 15:46, 10 March 2021
93Mar 16 19:45:27 mwmaint1002 mediawiki_job_parser_cache_purging[100664]: Deleting objects expiring before 19:45, 16 March 2021
94Mar 23 09:25:13 mwmaint1002 mediawiki_job_parser_cache_purging[120833]: Deleting objects expiring before 09:25, 23 March 2021
95Mar 30 08:56:02 mwmaint1002 mediawiki_job_parser_cache_purging[92844]: Deleting objects expiring before 08:56, 30 March 2021
96Apr 6 02:31:55 mwmaint1002 mediawiki_job_parser_cache_purging[185950]: Deleting objects expiring before 02:31, 6 April 2021
97Apr 13 00:41:29 mwmaint1002 mediawiki_job_parser_cache_purging[17162]: Deleting objects expiring before 00:41, 13 April 2021
98Apr 19 18:08:28 mwmaint1002 mediawiki_job_parser_cache_purging[151047]: Deleting objects expiring before 18:08, 19 April 2021
99Apr 26 02:20:46 mwmaint1002 mediawiki_job_parser_cache_purging[90978]: Deleting objects expiring before 02:20, 26 April 2021
100May 2 01:51:04 mwmaint1002 mediawiki_job_parser_cache_purging[216627]: Deleting objects expiring before 01:51, 2 May 2021
101May 7 05:23:37 mwmaint1002 mediawiki_job_parser_cache_purging[56878]: Deleting objects expiring before 05:23, 16 May 2021

Still running: May 10 04:23:37 mwmaint1002 mediawiki_job_parser_cache_purging[56878]: .......................] 12.45%

If the purge script is getting unreliable, should we look into that being related to our space issues?

It takes 4-5 days to run instead of <24h, and this has been the case for a few months. Overall though, it should not affect its reliability or the space consumed (other than having averaged retention around 32 days, between 30-35 days, instead of 30 days), but I'm not sure it would explain the more linear and continued growth we're seeing.

It might actually be correlated in the other direction in that having stored significantly more cache blobs, likely means it takes longer to iterate through them all with our fixed batch size. (Note this is speculation on my part, we have not yet determined the cause of the slowdown in the purge script.)

In any event, I am working already on improvements to make the purge script run faster, but probably by then it will have completed the run for this task, so I'll track that work elsewhere for now.

If the purge script is getting unreliable, should we look into that being related to our space issues?

It just takes more time to run, but it does delete the data, so I don't think it explains it.
Probably the reason for this is the amount of data we have (and the average size of the rows)

[mediawiki/extensions/DiscussionTools@master] Allow talk pages to have a different ParserCache expiry

https://gerrit.wikimedia.org/r/685180

@Esanders @DLynch This patch is ready for review.

Change 685878 merged by jenkins-bot:

[mediawiki/core@master] purgeParserCache.php: Remove carriage return printer, minor cleanup

https://gerrit.wikimedia.org/r/685878

Mentioned in SAL (#wikimedia-operations) [2021-05-18T12:40:29Z] <Krinkle> krinkle@mw1002 purge-parsercache-now.php on pc1010 (spare, depooled), ref P16060, T280605, T282761

  • Perform the reclaim sequence on the parser cache database servers.

The DBAs have begun this work as of yesterday. A few servers remaining, I will check back next Monday.

  • Perform the reclaim sequence on the parser cache database servers.

The DBAs have begun this work as of yesterday. A few servers remaining, I will check back next Monday.

hi @Krinkle – has any new information about the above surfaced since you last checked with the DBAs?


Note: I happened to be looking at this ticket when I saw the comment you posted in T280605#7099921 and thought to myself, "It's after 'next Monday,' I might as well ask while I'm here."

Mentioned in SAL (#wikimedia-operations) [2021-05-25T17:34:00Z] <Krinkle> mwmaint1002: Running purge-parsercache-now.php on server 2/4 (pc1007, depooled spare). Ref P16060, T280605, T282761.

Change 685181 merged by jenkins-bot:

[operations/mediawiki-config@master] Temporarily shorten $wgParserCacheExpireTime from 30 to 21 days

https://gerrit.wikimedia.org/r/685181

Confirmed via mwdebug1002, puring a page previously had:
"cachereport":{"origin":"mw1395", "timestamp":"20210512020440","ttl":2592000,
and now:
"cachereport":{"origin":"mwdebug1002","timestamp":"20210525180556","ttl":1814400,

where 1814400 is 21d*24h*3600s.

Change 685180 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Allow talk pages to have a different ParserCache expiry

https://gerrit.wikimedia.org/r/685180

Change 694314 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.7] Allow talk pages to have a different ParserCache expiry

https://gerrit.wikimedia.org/r/694314

Change 698201 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] Set wgDiscussionToolsTalkPageParserCacheExpiry to 10 days

https://gerrit.wikimedia.org/r/698201

Change 694314 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.7] Allow talk pages to have a different ParserCache expiry

https://gerrit.wikimedia.org/r/694314

Change 698201 merged by jenkins-bot:

[operations/mediawiki-config@master] Set wgDiscussionToolsTalkPageParserCacheExpiry to 10 days

https://gerrit.wikimedia.org/r/698201

Verified on mwdebug that:

  • Before this, articles on nlwiki have 21 days expiry (purge article, check New PP in source), as do talk pages.
  • After this, talk pages have 10 days instead, articles still 21 days (purge, check New PP again).

Mentioned in SAL (#wikimedia-operations) [2021-06-08T15:23:39Z] <Krinkle> mwmaint1002: Running purge-parsercache-now.php on server 4/4 (pc1009) ref P16060, T280605, T282761.

  • Reduce overall ParserCache expiry in wmf-config (reduce fom 30 days to 22 days). Prior art: T167784, T210992. – @Krinkle
  • Add ParserCache expiry override for talk pages in DiscussionTools extension (reduce from 30 days to 10 days). – @Krinkle, with code review by Editing-team.

The new DiscussionTools option has been deployed last week (thanks Ed), and the config changes have been deployed by me since as well. We're continuing to do the purging cycle. We have since the start of this acculumated more excess data so the purging is taking longer than anticipated. We're at the last server now per the above Stashbot log message.

Mentioned in SAL (#wikimedia-operations) [2021-06-10T10:28:51Z] <kormat> running optimize tables against pc1009 (pc3) T282761

Optimize of pc1009 (and replica) finished.

Disk space usage went from 3.9TB to 2TB.

Done from our side @ppelberg. We'll monitor over the next two weeks and report to the next task.

Done from our side @ppelberg. We'll monitor over the next two weeks and report to the next task.

Almost. I'll be putting pc3 back to it's production configuration on monday or so (i.e. moving pc1010 out, and making pc1009 the primary again). From that point we'll be back in a stable setup.

@Kormat don't forget to truncate pc1010 after a few days, to make sure we leave it clean and with only pc1 data.

Done from our side @ppelberg. We'll monitor over the next two weeks and report to the next task.

Excellent. We appreciate you continuing to keep us updated, @Krinkle


Note: I am assuming T280606 is the "next task" you are referring to in T280605#7151500. If this is NOT accurate, can you please comment as much?