Page MenuHomePhabricator

Database tables for checkwiki tool needs maintenance to prevent data loss
Closed, ResolvedPublic

Description

The checkwiki_p.cw_new and checkwiki_p.cw_dumpscan tables in the checkwiki tool's database are not currently being replicated because the recent toolsdb crash caused a mismatch in some replicated data.

In order to restore replication, we will need to find time to shut down the tool and reimport it.

Details

Event Timeline

s51080__checkwiki_p.cw_dumpscan is a temporary table . I have shutdown the program that uses this table and truncated the table.
s51080__checkwiki_p.cw_new is in continuous use logging recentchanges for 7 wikis via a cron job every 10 minutes. Need to schedule a time to shutdown the program using this table to minimize downtime.

I see I waited way too long to circle back to this task. It looks like dumpscan has writes in it again. Sorry about that.

I can help restore replication on this next week if you have time. I work in US Pacific Time and need maybe an hour at some point to do the re-import that needs to be done. I can also try to help early or late if need be. Let me know what might work next week (if that week works) for taking down things for a bit.

nskaggs moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.

s51080checkwiki_p.cw_dumpscan is not needed until 20 Aug when the next round of wiki dumps become available.
For s51080
checkwiki_p.cw_new I am available pretty much any time to shutdown the continuous service that is using it.

@Bamyers99 I think I misread your comments as "after August 20th", when I think you meant "before August 20th". I hope the timing hasn't been spoiled.

Feel free to poke on freenode #wikimedia-cloud during US daytime hours to coordinate.

Change 636469 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolsdb: remove temporarily replication filters

https://gerrit.wikimedia.org/r/636469

Bstorm claimed this task.

I removed the filters after restoring replication in T266587

Change 636469 merged by Bstorm:
[operations/puppet@production] toolsdb: remove temporary replication filters

https://gerrit.wikimedia.org/r/636469