Page MenuHomePhabricator

Switchover people.wikimedia.org - September 2023
Closed, ResolvedPublic

Description

Based on https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Manual_switch and https://wikitech.wikimedia.org/wiki/People.wikimedia.org#How_to_perform_a_switch-over_between_data_centers:

  • find rsync_src_host and rsync_dest_host in /hieradata/role/common/microsites/peopleweb.yaml
  • SSH to dst_host (you should see a warning in motd) and run manual rsync command:
    • rsync -avp rsync://people1004.eqiad.wmnet:/people-home /home/
  • Run puppet on dst_host
  • In puppet hieradata the rsync_src_host and rsync_dest_host hosts need to be flipped: https://gerrit.wikimedia.org/r/959690
    • merge and deploy change to both people machines
    • check motd on both servers (should be swapped)
  • The DNS discovery name peopleweb.discovery.wmnet needs to be switched from one backend to another: https://gerrit.wikimedia.org/r/959693
    • merge and deploy DNS change
  • confirm new backend gets traffic (tail -f /var/log/apache2/*.log).
  • Inform the users via a mailing list like "sre-at-large" (since that should cover shell users) that we switched

Event Timeline

LSobanski triaged this task as Medium priority.Sep 6 2023, 12:03 PM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.
Jelto updated the task description. (Show Details)
Jelto moved this task from Backlog to Work in Progress on the collaboration-services board.

Change 959690 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] peopleweb: switch rsync source and dest between eqiad and codfw

https://gerrit.wikimedia.org/r/959690

Change 959693 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/dns@master] switch peopleweb from eqiad to codfw

https://gerrit.wikimedia.org/r/959693

Change 959690 merged by Jelto:

[operations/puppet@production] peopleweb: switch rsync source and dest between eqiad and codfw

https://gerrit.wikimedia.org/r/959690

Change 959693 merged by Jelto:

[operations/dns@master] switch peopleweb from eqiad to codfw

https://gerrit.wikimedia.org/r/959693

Mentioned in SAL (#wikimedia-operations) [2023-09-25T09:38:36Z] <jelto> switch people.wikimedia.org to codfw - T345618

Jelto updated the task description. (Show Details)

people.wikimedia.org was switched to codfw. I'm closing this task.