As part of T123565 we need to do a data reload to re-index data for Geosearch. At the same time, we will do a full reinstall of wdqs1001 to enable use of new disk space.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Smalyshev | T123565 [EPIC] Support geo-coordinate search for WDQS | |||
Resolved | Gehel | T133566 Reinstall and data reload of WDQS servers | |||
Resolved | Smalyshev | T133986 Failure of wdqs-updater after data import | |||
Resolved | • Cmjohnson | T120712 install two Intel 320 Series SSDSA2CW300G3 2.5" 300GB each in wdqs1001/wdqs1002 | |||
Resolved | Smalyshev | T120714 implement wdqs1001/1002 disk upgrades (extend lvm) |
Event Timeline
Planned sequence:
- (day before) Send email to the wikidata list
- Take wdq1001 out of varnish config
- Shut down and reimage wdq1001. Verify disk partitioning is correct.
- Deploy new code from wdq-deploy repo. Do NOT restart wdq1002 yet!
- Reload data to wdq1001 from https://dumps.wikimedia.org/wikidatawiki/entities/20160425/ dump ttl-gz version (should be ready by then)
- Start updater on wdq1001 and wait for it to catch up
- Re-add wdq1001 to varnish, verify it's ready to serve requests
- Disable updater or wdq1002
- Put wdq1002 into maintenance mode (no need to take it out of varnish as we are only reloading data, not reimaging)
- Reload wdq1002 data from the same dump as above.
- Re-enable updater on wdq1002 and wait until it catches up
- Remove maintenance mode from wdq1002
- Verify everything works fine and queries run on both servers
- Send the victory email to wikidata
- PROFIT!
Change 285345 had a related patch set uploaded (by Gehel):
Depooled wdqs1001 during reinstall
Change 285353 had a related patch set uploaded (by Gehel):
Modify partitions to reflect new disk added in WDQS nodes
Change 285353 merged by Gehel:
Modify partitions to reflect new disk added in WDQS nodes
Mentioned in SAL [2016-04-26T09:50:10Z] <gehel> starting reinstall of wdqs1001 (T133566)
While rebuilding the RAID to add new disks, I realized wdqs1001 has 2x 300GB + 2x 150GB disks. I'm reinstalling anyway to ensure we don't run on a single node, but it does not look like what was planned in T119579 / T120712. i'll check with @RobH and/or @Cmjohnson when they arrive.
Change 285387 had a related patch set uploaded (by Gehel):
WDQS - Smaller /var/lib/wdqs partition
Change 285716 had a related patch set uploaded (by Gehel):
Revert "Depooled wdqs1001 during reinstall"
Mentioned in SAL [2016-04-27T20:09:09Z] <gehel> adding back wdqs1001 to varnish configuration after reinstall (T133566)
Mentioned in SAL [2016-04-27T20:32:13Z] <gehel> switching wdqs1002 to maintenance and reimporting data (T133566)
Mentioned in SAL [2016-04-28T14:32:14Z] <gehel> wdqs-updater started on wdqs1002 (T133566)