Page MenuHomePhabricator
Create Task
Maniphest T272512

Apply outstanding schema changes for "objectcache" tables in production (exptime, flags, modtoken)
Closed, ResolvedPublic
Actions

Description

  1. Alters to run:
  1. Where to run those changes:
    • Core MW tables of all wikis: all.dblist
  2. When to run those changes: At any time
  3. If the schema change is backwards compatible: No
  4. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster.
  5. if the data should be made available on the labs replicas and/or dumps: Not sure

This table is barely used and should not have much read and write on it (and be empty IIRC). It should be also truncated and locked before applying if it's not empty.

Progress:

  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8
  • pc1
  • pc2
  • pc3

Details

SubjectRepoBranchLines +/-
docs: Document how SqlBagOStuff::createTables() worksmediawiki/coremaster+14 -4
Avoid writing to objectcache tablemediawiki/extensions/Translatemaster+5 -4
upload: Migrate uploadstatus entry to MainObjectStash instead of dbmediawiki/coremaster+5 -3
Use MainStash instead of db-replicatedmediawiki/extensions/AbuseFilterwmf/1.38.0-wmf.18+1 -1
Use MainStash instead of db-replicatedmediawiki/extensions/AbuseFiltermaster+1 -1
page: Use MainObjectStash instead of 'db-replicated' cachemediawiki/corewmf/1.38.0-wmf.17+6 -6
page: Use MainObjectStash instead of 'db-replicated' cachemediawiki/coremaster+6 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ladsgroup added a comment.May 11 2021, 9:08 AM

I haven't had time to look at it with all of the craziness of the world but it has been on my (very long) todo list since February.

• Marostegui added a comment.May 11 2021, 9:20 AM

No worries, just double checking tasks we want/can prepare before the switchover :)

Krinkle subscribed.May 15 2021, 11:10 PM

I suggest waiting with this until https://gerrit.wikimedia.org/r/c/mediawiki/core/+/666780/24 can roll out at which point we can probably just do a one-time purge of whatever is in these tables (maybe in small groups of wikis, with a day in-between to reduce the load a bit).

Ref T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.

Once that work is done, we may also be moving some of these features to use MainStash (which will then be on x2 mysql instead of Redis), instead of objectcache on core tables (as done currently via CACHE_DB or 'db-replicated', instead of MainStash).

In proving and preparing for this, we actually moved some additional features from MainStash-Redis to CACHE_DB, but I think it would make sense to then move those further to MainStash.

Doing that will implicitly de-reference much of what's there anyway, at which point we can truncate whatever is left then. (Instead of truncating multiple times, once for the alter, and then again implicitly by moving the backend pointer to mainstash).

Ladsgroup mentioned this in T274174: Add modtoken field and flags to objectcache table.May 27 2021, 8:36 AM
Krinkle added a parent task: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Sep 20 2021, 7:14 PM
aaron subscribed.Sep 21 2021, 9:46 PM

I assume we'd want varbinary(14), so that it can store "infinity".

• Marostegui updated the task description. (Show Details)Sep 30 2021, 8:18 AM
• Marostegui removed • Marostegui as the assignee of this task.Oct 21 2021, 7:36 AM

Removing myself from this task, as this is waiting for @Ladsgroup to investigate when he's got time for it.

• Marostegui updated the task description. (Show Details)Oct 28 2021, 5:44 AM
Ladsgroup added a comment.Dec 20 2021, 4:06 PM

I looked at this.

It's three distinct problems (I admit I looked at frwiki only but for most places it should be the same).

  • echo seen
    • We can simply drop those, the high TTL to 2038 is fixed by I99230d2351b40751a3f2f5123c5f38693120259e and the values are not being read anymore as they have been migrated to kask. I can't find a way to see when it was actually set to mysql but it was possibly a blip. The funny thing is that function that creates the key says it's memcached but it was never on memcached as far as I can see.
  • :page-recent-delete: this is important and better not be deleted but I don't know why it's not migrated to mainstash.
  • I do see a lot of frwiki:twoColConflict_yourText:foo:123 and that definitely needs to be migrated to mainstash or memcached. I'll create a ticket for the team for it to take a look.

Does my assessment look correct @Krinkle?

Ladsgroup mentioned this in T298039: Migrate TwoColConflict usage of objectcache away from the table.Dec 20 2021, 4:33 PM
JJMC89 reopened subtask T298039: Migrate TwoColConflict usage of objectcache away from the table as Open.Dec 21 2021, 12:06 AM
Umherirrender mentioned this in T298206: Add objectcache.modtoken and objectcache.flags to wmf wikis.Dec 22 2021, 6:44 PM
Krinkle renamed this task from Alter objectcache.exptime to Apply outstanding schema changes for "objectcache" tables in production (exptime, flags, modtoken).Dec 22 2021, 6:51 PM
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle merged a task: T298206: Add objectcache.modtoken and objectcache.flags to wmf wikis.Dec 22 2021, 6:54 PM
Krinkle added a subscriber: Umherirrender.
Krinkle updated the task description. (Show Details)Dec 22 2021, 7:32 PM
Krinkle added a comment.Dec 22 2021, 7:37 PM

I added x2 to the list, but then removed it again. Upon checking I realize we hadn't yet created these tables on the x2 cluster in the first place (mainly because it's non-obvious which database name it would need to go under.) I'll handle that on the parent task and remove this as blocker for MainStash DB.

Krinkle removed a parent task: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 22 2021, 7:37 PM
thiemowmde closed subtask T298039: Migrate TwoColConflict usage of objectcache away from the table as Resolved.Jan 4 2022, 7:08 PM
Stashbot added a comment.Jan 5 2022, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-05T13:33:44Z] <Amir1> delete echo keys from objectchange in frwiki (T272512)

Ladsgroup added a comment.Jan 5 2022, 1:39 PM
In T272512#7598488, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2022-01-05T13:33:44Z] <Amir1> delete echo keys from objectchange in frwiki (T272512)

I cleaned it on all of s6 now.

Stashbot added a comment.Jan 10 2022, 10:16 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-10T10:16:22Z] <Amir1> removing echo objectcache entries on all wikis (T272512)

gerritbot added a comment.Jan 11 2022, 10:01 AM

Change 753003 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] page: Use MainObjectStash instead of 'db-replicated' cache

https://gerrit.wikimedia.org/r/753003

gerritbot added a project: Patch-For-Review.Jan 11 2022, 10:01 AM
gerritbot added a comment.Jan 17 2022, 11:12 PM

Change 753003 merged by jenkins-bot:

[mediawiki/core@master] page: Use MainObjectStash instead of 'db-replicated' cache

https://gerrit.wikimedia.org/r/753003

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.18; 2022-01-17).Jan 18 2022, 12:00 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 12:10 AM
gerritbot added a comment.Jan 18 2022, 9:10 AM

Change 754602 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.17] page: Use MainObjectStash instead of 'db-replicated' cache

https://gerrit.wikimedia.org/r/754602

gerritbot added a project: Patch-For-Review.Jan 18 2022, 9:10 AM
gerritbot added a comment.Jan 18 2022, 9:28 AM

Change 754602 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.17] page: Use MainObjectStash instead of 'db-replicated' cache

https://gerrit.wikimedia.org/r/754602

Stashbot added a comment.Jan 18 2022, 9:32 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-18T09:32:41Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.17/includes: Backport: [[gerrit:754602|page: Use MainObjectStash instead of 'db-replicated' cache (T272512)]] (duration: 00m 56s)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.17; 2022-01-10); removed MW-1.38-notes (1.38.0-wmf.18; 2022-01-17).Jan 18 2022, 10:00 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 10:10 AM
gerritbot added a comment.Jan 19 2022, 5:51 PM

Change 755450 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/AbuseFilter@master] Use MainStash instead of db-replicated

https://gerrit.wikimedia.org/r/755450

gerritbot added a project: Patch-For-Review.Jan 19 2022, 5:51 PM
gerritbot added a comment.Jan 20 2022, 12:03 AM

Change 755450 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] Use MainStash instead of db-replicated

https://gerrit.wikimedia.org/r/755450

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2022, 12:10 AM
ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.19; 2022-01-24); removed MW-1.38-notes (1.38.0-wmf.17; 2022-01-10).Jan 20 2022, 1:00 AM
gerritbot added a comment.Jan 24 2022, 11:27 AM

Change 756083 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/AbuseFilter@wmf/1.38.0-wmf.18] Use MainStash instead of db-replicated

https://gerrit.wikimedia.org/r/756083

gerritbot added a project: Patch-For-Review.Jan 24 2022, 11:27 AM
gerritbot added a comment.Jan 24 2022, 3:28 PM

Change 756083 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@wmf/1.38.0-wmf.18] Use MainStash instead of db-replicated

https://gerrit.wikimedia.org/r/756083

Stashbot added a comment.Jan 24 2022, 3:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T15:48:40Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.18/extensions/AbuseFilter/includes/ServiceWiring.php: Backport: [[gerrit:756083|Use MainStash instead of db-replicated (T272512)]] (duration: 00m 49s)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.18; 2022-01-17); removed MW-1.38-notes (1.38.0-wmf.19; 2022-01-24).Jan 24 2022, 4:00 PM
gerritbot added a comment.Feb 16 2022, 2:31 PM

Change 763240 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] upload: Migrate uploadstatus entry to MainObjectStash instead of db

https://gerrit.wikimedia.org/r/763240

gerritbot added a comment.Feb 16 2022, 4:08 PM

Change 763240 merged by jenkins-bot:

[mediawiki/core@master] upload: Migrate uploadstatus entry to MainObjectStash instead of db

https://gerrit.wikimedia.org/r/763240

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.23; 2022-02-21); removed MW-1.38-notes (1.38.0-wmf.18; 2022-01-17).Feb 16 2022, 5:00 PM
gerritbot added a comment.Feb 28 2022, 7:43 AM

Change 766559 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Translate@master] Avoid writing to objectcache table

https://gerrit.wikimedia.org/r/766559

gerritbot added a comment.Feb 28 2022, 12:37 PM

Change 766559 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Avoid writing to objectcache table

https://gerrit.wikimedia.org/r/766559

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.24; 2022-02-28); removed MW-1.38-notes (1.38.0-wmf.23; 2022-02-21).Feb 28 2022, 1:00 PM
Ladsgroup moved this task from Unclear/blocked to proceed to Backlog on the Schema-change-in-production board.Mar 7 2022, 9:51 AM

The table is fully cleaned now. Nothing writes to it anymore in any wiki.

Ladsgroup moved this task from Blocked to Ready on the DBA board.Mar 7 2022, 9:52 AM
• Marostegui changed the task status from Stalled to Open.Mar 9 2022, 8:35 AM

@Ladsgroup confirm this is good to go?
I have checked the table on frwiki and it is indeed now empty (and same for parsercache)

• Marostegui added a comment.Mar 9 2022, 8:37 AM

We'd also need the ALTER for flags and modtoken columns.

Ladsgroup added a comment.Mar 9 2022, 8:39 AM

I don't know about ParserCache table. That's another beast I don't want to tackle atm. But objectcache should be fine to go.

• Marostegui added a comment.Mar 9 2022, 8:41 AM

What do you mean?
As far as I can see we need to alter objectcache table on sX and on pcX - not the pcXXX tables.

Ladsgroup added a comment.Mar 9 2022, 8:43 AM

Why do we have objectcache table on pcX? Which wiki they belong to? Why our infra is so inconsistent 😭

I need to check those (and maybe we can drop the table there?) but for sX it should be fine.

• Marostegui added a comment.Mar 9 2022, 8:45 AM
In T272512#7763137, @Ladsgroup wrote:

Why do we have objectcache table on pcX? Which wiki they belong to? Why our infra is so inconsistent 😭

You tell me :-)
So that one shouldn't be altered?

I need to check those (and maybe we can drop the table there?) but for sX it should be fine.

I will leave it out for now then
But we need to get the column definition for flags and modtoken.

Ladsgroup added a comment.Mar 9 2022, 10:02 AM

This would be it: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/666780/26/maintenance/archives/patch-objectcache-modtoken.sql

• Marostegui updated the task description. (Show Details)Mar 9 2022, 10:19 AM
• Marostegui claimed this task.Mar 9 2022, 10:22 AM

I will work on this one too.
Altered s6 (all wikis). Giving it 24h to settle just in case

• Marostegui updated the task description. (Show Details)Mar 9 2022, 10:22 AM
• Marostegui moved this task from Backlog to In progress on the Schema-change-in-production board.
• Marostegui moved this task from Ready to In progress on the DBA board.
Stashbot added a comment.Mar 9 2022, 10:29 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-09T10:29:31Z] <marostegui> dbmaint on s6@eqiad T272512

• Marostegui updated the task description. (Show Details)Mar 9 2022, 1:29 PM
Stashbot added a comment.Mar 9 2022, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-09T13:47:03Z] <marostegui> dbmaint on s8@eqiad T272512

gerritbot added a comment.Mar 9 2022, 4:02 PM

Change 769461 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] docs: Document how SqlBagOStuff::createTables() works

https://gerrit.wikimedia.org/r/769461

Krinkle added a comment.EditedMar 9 2022, 4:10 PM

The core objectcache tables, parsercache table shards, and mainstash DB are all backed by the same schema and SqlBagOStuff interface. Having said that, the new features for which this particular schema addition was drafted are unlikely to be needed for ParserCache any time soon so it's certainly fine to post-pone that for now. Although for consistency, transparency, and reduced code complexity it would make sense to apply "eventually". The longer we wait, the more risk from accidental misunderstandings or assumptions about what the state of any given table is.

The pc% tables were originally, or intended to be, initialised by SqlBagOStuff::createTables(). I've just pushed a comment to document this more clearly. It works by running CREATE TABLE pc123 LIKE objectcache; after running the standard objectcache.sql on each database. However, it seems we no longer have that file. And yes this means each pc database has an empty objectcache table that is simply an empty template. This is not required or useful now, but it is a known and expected innocent side-effect.

• Marostegui added a comment.Mar 9 2022, 4:13 PM

Sure, it can be done easily as they are empty. We should research though if they can or need to be deleted in order to also reduce confusion like the one we had on this ticket.

• Marostegui updated the task description. (Show Details)Mar 10 2022, 5:43 AM
Stashbot added a comment.Mar 10 2022, 5:45 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T05:45:51Z] <marostegui> dbmaint on s2@eqiad T272512

Stashbot added a comment.Mar 10 2022, 5:45 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T05:45:56Z] <marostegui> dbmaint on pc1@eqiad T272512

Stashbot added a comment.Mar 10 2022, 5:45 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T05:45:58Z] <marostegui> dbmaint on pc2@eqiad T272512

Stashbot added a comment.Mar 10 2022, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T05:46:01Z] <marostegui> dbmaint on pc3@eqiad T272512

Stashbot added a comment.Mar 10 2022, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T05:46:03Z] <marostegui> dbmaint on s4@eqiad T272512

• Marostegui updated the task description. (Show Details)Mar 10 2022, 5:46 AM
Stashbot added a comment.Mar 10 2022, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T05:46:46Z] <marostegui> dbmaint on s5@eqiad T272512

• Marostegui updated the task description. (Show Details)Mar 10 2022, 6:03 AM
• Marostegui updated the task description. (Show Details)
Stashbot added a comment.Mar 10 2022, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T06:05:16Z] <marostegui> dbmaint on s7@eqiad T272512

Stashbot added a comment.Mar 10 2022, 6:07 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-10T06:07:40Z] <marostegui> dbmaint on s3@eqiad T272512

• Marostegui updated the task description. (Show Details)Mar 10 2022, 6:09 AM
• Marostegui closed this task as Resolved.Mar 10 2022, 10:57 AM
• Marostegui updated the task description. (Show Details)

All done, including parsercache

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 10 2022, 11:15 AM
gerritbot added a comment.Mar 25 2022, 12:54 AM

Change 769461 merged by jenkins-bot:

[mediawiki/core@master] docs: Document how SqlBagOStuff::createTables() works

https://gerrit.wikimedia.org/r/769461

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.5; 2022-03-28).Mar 25 2022, 1:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL