gerrit migration plan/steps
ActivePublic
Actions

Authored by Dzahn on May 5 2023, 6:15 PM.

Tags

None

Referenced Files

	F36991585: gerrit migration plan/steps
	May 10 2023, 8:38 PM

	F36991454: gerrit migration plan/steps
	May 10 2023, 6:05 PM

	F36991446: gerrit migration plan/steps
	May 10 2023, 6:02 PM

	F36991429: gerrit migration plan/steps
	May 10 2023, 5:47 PM

	F36991424: gerrit migration plan/steps
	May 10 2023, 5:44 PM

	F36991419: gerrit migration plan/steps
	May 10 2023, 5:39 PM

	F36991415: gerrit migration plan/steps
	May 10 2023, 5:37 PM

	F36991342: gerrit migration plan/steps
	May 10 2023, 4:38 PM

View All 11 Files

Subscribers

LSobanski

thcipriani

Tokens

"Like" token, awarded by thcipriani.

	# schedule and announce downtime
	# on gerrit1001: shortly before the scheduled downtime:
	# on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /var/lib/gerrit2/review_site/ rsync://gerrit1003.wikimedia.org/gerrit-var-lib/
	# on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /srv/gerrit/ rsync://gerrit1003.wikimedia.org/gerrit-data/
	# on gerrit1003: rsync -avp /srv/gerrit/plugins/lfs/ /srv/gerrit/data/lfs/
	# on gerrit1003: chown -R gerrit2:gerrit2 /var/lib/gerrit2
	# on gerrit1003: chown -R gerrit2:gerrit2 /srv/gerrit
	# scheduled downtime begins / IRC announcement
	# on cumin1001:sudo cookbook sre.hosts.downtime -r 'maintenance' -D 30 gerrit1001.wikimedia.org
	# on cumin1001:sudo cookbook sre.hosts.downtime -r 'maintenance' -H 1 gerrit1003.wikimedia.org
	# on icinga.wikimedia.org - manually schedule downtime for the checks connected to virtual server "gerrit.wikimedia.org". The cookbook does not find this virtual host.
	# on gerrit1003: disable puppet; stop gerrit? (sudo disable-puppet 'gerrit maintenance'; systemctl stop gerrit)
	# merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org - in web UI of gerrit(-old)
	# run authdns-update on ns0.wikimedia.org, see the diff but do NOT commit yet
	# on gerrit1001: disable puppet; stop gerrit! (sudo disable-puppet 'gerrit maintenance'; systemctl stop gerrit)
	# on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /var/lib/gerrit2/review_site/ rsync://gerrit1003.wikimedia.org/gerrit-var-lib/
	# on gerrit1001, as root, in a screen: rsync -avp --delete --bwlimit=100m /srv/gerrit/ rsync://gerrit1003.wikimedia.org/gerrit-data/
	# on gerrit1003: rsync -avp /srv/gerrit/plugins/lfs/ /srv/gerrit/data/lfs/
	# on gerrit1003: chown -R gerrit2:gerrit2 /var/lib/gerrit2
	# on gerrit1003: chown -R gerrit2:gerrit2 /srv/gerrit
	# on gerrit1003: start gerrit
	# say "yes" to authdns-update and actually merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org
	# wait 5 minutes
	# ..test https (https://gerrit.wikimedia.org in browser)
	# ..test ssh (e.g. ssh dzahn@gerrit-new.wikimedia.org -p 29418)
	# announce downtime is over
	# ensure gerrit1001 has puppet disabled and/or services are masked
	# grace period (how long?)
	# decom old host -> https://phabricator.wikimedia.org/T336427

Event Timeline

Dzahn created this paste.May 5 2023, 6:15 PM

Dzahn added a subscriber: thcipriani.

thcipriani awarded a token.May 5 2023, 6:25 PM

Dzahn edited the content of this paste. (Show Details)May 5 2023, 6:31 PM

Dzahn mentioned this in T326368: gerrit1003 service implementation task.May 5 2023, 6:36 PM

Dzahn edited the content of this paste. (Show Details)May 5 2023, 6:38 PM

Dzahn edited the content of this paste. (Show Details)May 10 2023, 4:38 PM

confirmed we have gigabit connection and upped bwlimit to 100m, so ten/twenty times faster now

updated regarding DNS change: now there is first step 9 and 10 to merge, other things, and then step 17 to actually deploy the DNS change.

Dzahn edited the content of this paste. (Show Details)May 10 2023, 5:44 PM

addd decom ticket, removed "raise bwlimit" question, added question how long the grace period should be (roughly). added test command for ssh

Dzahn edited the content of this paste. (Show Details)May 10 2023, 6:02 PM

Dzahn edited the content of this paste. (Show Details)May 10 2023, 6:05 PM

steps we have to add:

move/copy lfs data which is in new location on new host (for a good reason) (https://gerrit.wikimedia.org/r/c/operations/puppet/+/911363)
disable gerrit1001 and enable gerrit1003 as the source of replication to gerrit2002 and github: https://gerrit.wikimedia.org/r/c/operations/puppet/+/918589

Dzahn edited the content of this paste. (Show Details)May 10 2023, 8:38 PM

Need to add gerrit-old as a certificate SAN.

follow-ups later:

delete old lfs data from /root on gerrit1003
add gerrit-old to SAN
make puppet patch so that puppet can be re-enabled on gerrit1001 WITHOUT starting the gerrit service (mask it to be sure)
re-enable puppet on gerrit1001 so it doesn't fall out of puppetdb
wait for grace period (30 days)
remove old IP from homer repo and deploy to switches again
remove old IP from puppet (and CI?) repos
remove gerrit1001 from puppet repo
run decom cookbook on gerrit1001
give hardware back to dcops
give public IP back to netbox

gerrit migration plan/stepsActivePublicActions

Event Timeline

gerrit migration plan/steps
ActivePublic
Actions