Page MenuHomePhabricator

Migrate Wikilabels from labsdb1004 to clouddb1002
Closed, ResolvedPublic

Description

We'll need to perform a full dump and load of the wikilabels database from one (hardware) host to a virtual host.

We'll need two databases:

  • u_wikilabels
  • u_wikilabels_test

Steps

Once the database is primed (accounts and databases are prepared), we'll need to load our data in, run some tests, and then lock wikilabels in maintenance mode for a switchover.

1. Staging test

Copy the old u_wikilabels_test to the new u_wikilabels_test. Update the configuration to point to clouddb1002.clouddb-services.eqiad.wmflabs rather than "wikilabels-database". Check to see if everything works as intended. If everything worked as intended, we're ready to set a maintenance window.

2. Put Wikilabels into maintenance mode

Announce downtime for Wikilabels. 30 minutes should be sufficient. Create a deployment config change that adds a notice. See https://github.com/wikimedia/wikilabels-wmflabs-deploy/commit/6930d095413b3dc0e05eece9d9925b6557bd01de#diff-77ae19360cfaa1aaaea7086fab249bf4 for an example commit that adds a maintenance notice.

3. Do the maintenance

When the time comes, stop the service and confirm that a browser returns the right http code. Dump the u_wikilabels database and load it on the new host. Deploy the config change pointing toward clouddb1002.clouddb-services.eqiad.wmflabs and restart the service.

Confirm that everything works and declare victory.

Event Timeline

Halfak created this task.Mar 8 2019, 8:56 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)Mar 8 2019, 9:26 PM
Halfak updated the task description. (Show Details)

I've successfully deployed the new credentials to staging and restarted the service. It seems like everything is working as intended.

We're ready to schedule a maintenance window. I'm traveling and sick this week (oof), so I'd like to delay that for next. Sorry @Bstorm, I know we'd hoped to get this worked out this week.

I've exported pintoch.bz2 tobias47n9e.bz2 u_halfak.bz2 u_shiladsen.bz2 wikimaps_atlas.bz2 from labsdb1004. Only u_shiladsen has a lot of data in it. I figure that way I should be able to copy them over and import if you want. Otherwise, perhaps they are just things to archive somewhere?

Halfak added a subscriber: Shilad.Mar 18 2019, 2:07 PM

@Shilad ^ any opinions on this database?

Halfak added a subscriber: Harej.Mar 18 2019, 2:07 PM

I've just pushed the maintenance notice out to Wikilabels. @Harej posted a bunch of announcements. We're ready to try the migration at 1400 UTC tomorrow (March 19th)

@Shilad ^ any opinions on this database?

Wipe it! I no longer need it, but thanks for checking in!

OK the maintenance has been completed without an issue. That means we're ready to resolve this task. I'd still like to be present when labsdb1004 is taken offline just to make sure everything goes nicely on our end. But at this point, I don't expect a hiccup when it goes offline.

Halfak closed this task as Resolved.

Change 499910 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wikilabels: Update toolschecker to monitor the live DB

https://gerrit.wikimedia.org/r/499910

Change 499910 merged by Bstorm:
[operations/puppet@production] wikilabels: Update toolschecker to monitor the live DB

https://gerrit.wikimedia.org/r/499910