Hey folks, is this on track to happen? Anything else you need from our side?

TODO:

• Make pc1009 the pc3 primary again: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/700026
• Run stop slave; reset slave all on pc1009, so it stops replicating from pc1010
• Move pc1010 back to pc1: TODO
• Truncate pc1010
• Set pc1010 replicating from pc1007

All done.

Ohh, right, of course. Never mind then :)

In T280605#7151500, @Krinkle wrote:

Done from our side @ppelberg. We'll monitor over the next two weeks and report to the next task.

Optimize of pc1009 (and replica) finished.

Jun 8th: ~28h
Jun 9th: ~15.5h
Jun 10th: still running

In T284650#7146477, @Trizek-WMF wrote:

We can reuse former campaigns we created, if they are still accurate.

Can we know if any changes have been made to the s3 cluster since 2019-09-24? Any wikis added or removed from the list?

Yes please to both of those :)

Procedure looks good :)

Optimize of pc1008 (and replica) finished.

db1157 had a clean mysqlcheck run, repooling it now.

db1157 upgraded to buster. Running mysqlcheck now.

Re-labelling not necessary, as it wasn't re-labelled away from db1125 in the first place: T283300

Run finished at 2021-06-05T14:30. Running optimize over all pc* tables now.

15:39:55 <Krinkle> kormat: it's running now, tee'ed to /home/krinkle/purge_parsercache_now_pc1008.log

In T283131#7131272, @Marostegui wrote:

s6 hasn't given any issues, so maybe we can start working on this next week (after 3 weeks since we switched s6) and attempt to do the switchover the 17th?
@Kormat thoughts?

It's back in tendril+zarcillo, and is a replica of db1124.

pc1010 is now pc2 primary, and is no longer replicating from pc1008:

Current status:

• db1125 has been renamed, wiped, and reimaged
• It still needs to be re-added to tendril/zarcillo, and have an s4 snapshot deployed on it.

Example for https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/696454

In T283793#7125719, @Marostegui wrote:

Ah yes, I misunderstood you. Yes, indeed, that's why we run check_private_data after data sanitization on new wikis, so we can also get those private tables deleted.

In T283793#7125715, @Marostegui wrote:

Normally what we do is: redact_sanitarium.sh -d wikidatawiki -S socket_path | mysql -S socket_path wikidatawiki

In T283793#7124520, @Marostegui wrote:

See email - s8 reported some tables that need to be dropped

redact_sanitarium.sh completed, and a quick check showed it had been successful.

Status:

• Data copy from db2082 completed.
• mysql_upgrade ran
• redact_sanitarium.sh currently running.

Example for https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/696454

Current status:

• pc1 is repooled and back in service.
• pc1010 is now in pc2, and replicating from pc1008. This means it will have at least _some_ relevant entries when it becomes pc2 primary next week.

Running:
sudo transfer.py --type file --no-compress --no-encrypt --no-checksum db2082.codfw.wmnet:/srv/sqldata db2094.codfw.wmnet:/srv/sqldata.s8

db2082 is db2094:s8's master:

root@db2082.codfw.wmnet[(none)]> stop slave;
Query OK, 0 rows affected (0.036 sec)

Optimize of pc1007 (and replicas) finished.

🎉

In T283580#7111850, @Marostegui wrote:

We don't receive gerrit updates on the channel, do we?

The purge has finished as of 2021-05-26T06:00Z. I'll start the optimize process now.

I wouldn't say it's very _urgent_, but it would definitely be nice to have it done before the dc switchover preperations, for sanity-checking that circular replication is set up correctly.

I'd vote for a non-'bot'-specific channel name (so -bulk or -firehose or similar). Other than that, LGTM.

Current status:

• pc1010 is now the primary for pc1
• I've run stop slave on pc1010, so it no longer replicates from pc1007
• I've created a downtime for 7 days for pc[2007,2010].codfw.wmnet,pc1007.eqiad.wmnet

For posterity, here's the script i used for the heartbeat changes:

In T252528#7103046, @LSobanski wrote:

In T252528#7100826, @Kormat wrote:

This does mean that pt-heartbeat-wikimedia needs to be started manually after a boot, however.

@Kormat is this captured somewhere in documentation?

The grant for cumin2002 should now be fully deployed.

Optimize of pc1010 finished.

Yes indeed 🎉

This is now fixed. Puppet will no longer start/stop heartbeat. That is managed by db-switchover when changing masters. This does mean that pt-heartbeat-wikimedia needs to be started manually after a boot, however.

Deployment complete:

kormat@cumin1001:~(0:0)$ sudo debdeploy deploy -u 2021-05-20-wmfmariadbpy.yaml -Q C:wmfmariadbpy
Rolling out wmfmariadbpy:
Non-daemon update, no service restart needed

In T283228#7100571, @Kormat wrote:

pt-heartbeat-wikimedia fails to start on db2093 with:

Heartbeat restarted on all primaries.

pt-heartbeat-wikimedia fails to start on db2093 with:

DBD::mysql::st execute failed: Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.

This is due to the unusual config of the dbinventory section.