ParserCache is not deleting old rows after three months past the expiry in the secondary datacenter
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Jan 3 2025, 1:07 PM

Description

This is some rows from parsercache (this random thing is pc3):

cumin2024@pc1015.eqiad.wmnet[(none)]> select keyname, exptime from parsercache.pc106 order by keyname limit 100;
+-----------------------------------------------------+---------------------+
| keyname                                             | exptime             |
+-----------------------------------------------------+---------------------+
...
| abwiki:parsoid-pcache:idoptions:13198               | 2024-10-03 11:06:10 |
| abwiki:parsoid-pcache:idoptions:16053               | 2024-09-26 17:50:44 |
| abwiki:parsoid-pcache:idoptions:16205               | 2024-10-01 23:19:40 |
| abwiki:parsoid-pcache:idoptions:16647               | 2024-10-03 10:36:56 |
| abwiki:parsoid-pcache:idoptions:17529               | 2024-10-03 10:19:23 |
| abwiki:parsoid-pcache:idoptions:17654               | 2024-10-20 23:34:38 |
| abwiki:parsoid-pcache:idoptions:18150               | 2024-10-09 00:20:58 |
| abwiki:parsoid-pcache:idoptions:19164               | 2024-10-04 19:48:12 |
| abwiki:parsoid-pcache:idoptions:22184               | 2024-09-30 10:26:43 |
| abwiki:parsoid-pcache:idoptions:2663                | 2024-10-15 09:06:29 |
| abwiki:parsoid-pcache:idoptions:29851               | 2024-09-27 22:12:00 |
| abwiki:parsoid-pcache:idoptions:33474               | 2024-10-11 18:34:25 |
| abwiki:parsoid-pcache:idoptions:35170               | 2024-10-09 08:39:40 |
| abwiki:parsoid-pcache:idoptions:39506               | 2024-10-12 06:26:53 |
| abwiki:parsoid-pcache:idoptions:40441               | 2024-10-11 23:06:16 |
| abwiki:parsoid-pcache:idoptions:40694               | 2024-09-29 08:49:40 |
| abwiki:parsoid-pcache:idoptions:43771               | 2024-10-20 12:57:25 |
| abwiki:parsoid-pcache:idoptions:5105                | 2024-10-12 12:48:52 |
| abwiki:parsoid-pcache:idoptions:8054                | 2024-10-16 07:39:29 |
| abwiki:parsoid-pcache:idoptions:8325                | 2024-10-16 07:56:19 |
...

There are a lot more. Most of these should have been dropped three months ago (right after the expiry has passed)

Event Timeline

Ladsgroup created this task.Jan 3 2025, 1:07 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2025, 1:07 PM

lol because it's the secondary dc and the cleaner script is not being run there and I assume these entries are not in codfw so they didn't get cleaned up by replication.

Codfw parsercache is properly clean.

Ladsgroup renamed this task from ParserCache is not deleting old rows after three months past the expiry to ParserCache is not deleting old rows after three months past the expiry in the secondary datacenter.Jan 3 2025, 1:09 PM

The easiest way would be to allow multi-dc runs in modules/profile/files/mediawiki/maintenance/mw-cli-wrapper.py. I'll go bother serviceops.

In T382948#10429035, @Ladsgroup wrote:

lol because it's the secondary dc and the cleaner script is not being run there and I assume these entries are not in codfw so they didn't get cleaned up by replication.

Codfw parsercache is properly clean.

I think it maybe because of all the changes in replication and masters during the outages. That is my guess looking at the dates. I was on sabbatical so I am not 100% sure, but it could be related to those changes.

oh it can be. I think I should do a one-off clean up and then check later to see if it shows up again. Does that sound good to you @Marostegui ?

In T382948#10429455, @Ladsgroup wrote:

oh it can be. I think I should do a one-off clean up and then check later to see if it shows up again. Does that sound good to you @Marostegui ?

Go for it!

Marostegui assigned this task to Ladsgroup.Jan 7 2025, 12:47 PM

Marostegui moved this task from Triage to Ready on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2025-01-07T13:30:17Z] <Amir1> running mwscript purgeParserCache.php --wiki=aawiki --tag pc5 --age=2592000 --msleep 200 in *eqiad* (T382948)

I wrote an ad-hoc deletion bash. Running in a screen in cumin1002. It'll take a while to finish in pc3 (around 300M rows). Once that's done, I'll start checking and cleaning other sections.

Marostegui moved this task from Ready to In progress on the DBA board.Jan 8 2025, 8:25 AM

20% done. I'll take a look again in a week.

pc3 in eqiad is cleaned now but I think there are a couple more.

Running on pc5 (pc1014)

Mentioned in SAL (#wikimedia-operations) [2025-01-13T09:41:20Z] <Amir1> dbmaint on pc5@eqiad (T382948)

It seems it indeed was cloning issue. It's fully clean now:

ladsgroup@cumin1002:~$ sudo db-mysql pc1017 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$ sudo db-mysql pc1016 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$ sudo db-mysql pc1015 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$ sudo db-mysql pc1014 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$ sudo db-mysql pc1013 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$ sudo db-mysql pc1012 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$ sudo db-mysql pc1011 -e "select keyname, exptime from parsercache.pc225 where exptime < '2025-01-01' limit 20;"
ladsgroup@cumin1002:~$

Yeah, I've been truncating things for the new sections so it makes sense it is all clean now.

Maintenance_bot moved this task from In progress to Done on the DBA board.Jan 21 2025, 8:29 AM

ParserCache is not deleting old rows after three months past the expiry in the secondary datacenterClosed, ResolvedPublicActions

Description

Event Timeline

ParserCache is not deleting old rows after three months past the expiry in the secondary datacenter
Closed, ResolvedPublic
Actions