Page MenuHomePhabricator
Create Task
Maniphest T307142

bring new gitlab hardware servers into production
Closed, ResolvedPublic
Actions

Description

So far all gitlab* machines have been virtual.

Now we just got the first dedicated hardware, physical servers for gitlab.

There is one ticket for codfw and one for eqiad, both include gitlab* and gitlab-runner* machines.

codfw - T301183
eqiad - T301177

Both are now ready for us to take over.

GitLab Runner migration

gitlab-runner hosts can be integrated independently from GitLab migration. So the following machines need role(gitlab_runner):

  • gitlab-runner1002.eqiad.wmnet (paused)
  • gitlab-runner1003.eqiad.wmnet (paused)
  • gitlab-runner1004.eqiad.wmnet (paused)
  • gitlab-runner2002.codfw.wmnet (paused)
  • gitlab-runner2003.codfw.wmnet (paused)
  • gitlab-runner2004.codfw.wmnet (paused)

If the above Runners are configured and ready, ganeti VMs gitlab-runner1001.eqiad.wmnet and gitlab-runner2001.codfw.wmnet can be unregistered and destroyed.

  • decommission gitlab-runner1001.eqiad.wmnet
  • decommission gitlab-runner2001.codfw.wmnet

GitLab migration

GitLab migration needs some additional preparation.

  • register second service IPs for gitlab1003
  • validate puppet code and GitLab configuration with a physical replica on gitlab1003 (also bullseye)
  • evaluate additional configuration changes for potential HA setups T323201
  • create custom partman config for GitLab 793534
    • bigger / root volume
    • dedicated /srv volume (and move backups back to this folder instead of /mnt)
    • dedicated Docker volume not needed
    • dedicated Registry volume (see gitlab_rails['registry_path'])

See checklist for replica migration: T307142#7969993
See checklist for production migration: T307142#7971192

Tasks after downtime:

  • switch bacula fileset for gitlab from /mnt to /srv 800357
  • increase TTL for DNS records
  • check bacula backups for new host next day
  • migrate additional hosts
    • gitlab2002.wikimedia.org as replica T296713
    • gitlab2003.wikimedia.org as replica see T323201
  • decommission old hosts
    • gitlab2001.wikimedia.org
    • gitlab1001.wikimedia.org
    • remove dns entries
    • remove hosts from hiera

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+6 -0gitlab: add reserved service IP 208.80.153.8, point to replica-new
operations/dnsmaster+0 -4wikimedia.org: remove gitlab-old.wikimedia.org
operations/puppetproduction+0 -8gitlab: remove gitlab1001 from Hiera
operations/puppetproduction+4 -9site/gitlab: remove gitlab1001, update comments
operations/puppetproduction+2 -7site: remove gitlab1001, adjust gitlab machine descriptions
operations/puppetproduction+0 -5DHCP: remove gitlab1001
operations/puppetproduction+1 -2gitlab/acme_chief: remove gitlab1001 from list of (passive) hosts
operations/puppetproduction+0 -6site: remove gitlab2001
operations/dnsmaster+0 -4wikimedia.org: remove gitlab-replica-old.wikimedia.org
operations/puppetproduction+1 -16site/hiera: remove gitlab2001 after decom
operations/puppetproduction+1 -1gitlab: switch gitlab2001 back to "insetup" role
operations/puppetproduction+1 -2gitlab/acme_chief: remove gitlab2001 from list of (passive) hosts
operations/puppetproduction+0 -5DHCP: remove gitlab2001
operations/puppetproduction+0 -10site: remove old gitlab runners
operations/puppetproduction+0 -6install_server: remove gitlab-runner 2001
operations/puppetproduction+0 -6install_server: remove gitlab-runner1001
operations/puppetproduction+20 -20logtash: replace gitlab1001 with gitlab1004 in tests
operations/puppetproduction+3 -3gitlab: make gitlab1003 new replica
operations/dnsmaster+8 -8wikimedia.org: make gitlab1003 the new gitlab-replica
operations/dnsmaster+4 -4wikimedia.org: reduce TTL for gitlab A and AAAA to 5m
operations/puppetproduction+5 -4gitlab: make gitlab1004 new production instance
operations/dnsmaster+8 -8wikimedia.org: make gitlab1004 the new gitlab production host
operations/puppetproduction+15 -19gitlab: separate hiera config for gitlab hosts
operations/puppetproduction+22 -20gitlab: allow multiple passive hosts
operations/dnsmaster+2 -0wikimedia.org: add gitlab-replica-new records
operations/puppetproduction+12 -2gitlab: use gitlab1003 as replia/passive host
operations/puppetproduction+6 -1acme_chief: add new gitlab hosts to acl for gitlab certificate
operations/puppetproduction+11 -0aptrepo: add gitlab package for bullseye
operations/puppetproduction+9 -2gitlab: fix gitlab-ce apt component on bullseye hosts
operations/puppetproduction+1 -1idp: add gitlab-replica-new to idp
operations/dnsmaster+2 -0dns: add PTR records for gitlab-replica-new.wikimedia.org
operations/puppetproduction+86 -1install_server: add custom partman config for gitlab
operations/puppetproduction+7 -1gitlab: move backups back to /srv/gitlab-backup for physical hosts
operations/dnsmaster+4 -0wikimedia.org: add gitlab-new records + PTR
operations/dnsmaster+4 -0wikimedia.org: move gitlab-replica from netbox to dns repo
operations/dnsmaster+4 -0wikimedia.org: move gitlab from netbox to dns repo
operations/puppetproduction+1 -1idp: add gitlab-new to idp
operations/puppetproduction+8 -3gitlab: use gitlab1004 as replia/passive host
operations/puppetproduction+10 -16gitlab: rsync config and data backup to same folder on replica
operations/puppetproduction+29 -1gitlab: create role/profile to temp use gitlab1004 to store backups
operations/puppetproduction+3 -3site/gitlab: add gitlab-runner role also on eqiad physical machines
operations/puppetproduction+3 -7site: use gitlab-runner role on all new physical gitlab-runner servers
operations/puppetproduction+5 -1add gitlab-runner role on new physical server gitlab-runner2002
Show related patches Customize query in gerrit
ReferenceSource BranchDest BranchAuthorTitle
repos/releng/gitlab-settings!11physical-hostsmainjeltoadd settings files for new physical hosts
Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 31 2022, 10:20 AM

Change 801652 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/dns@master] wikimedia.org: make gitlab1003 the new gitlab-replica

https://gerrit.wikimedia.org/r/801652

Jelto added a comment.EditedMay 31 2022, 12:14 PM

Checklist for todays gitlab-replica migation from gitlab2001 to gitlab1003:

Preparations before downtime:

  • register second service IPs for gitlab1003
  • validate puppet code and GitLab configuration with a physical replica on gitlab1003 (also bullseye)
  • move gitlab-replica.wikimedia.org entry from netbox to dns repo 800709
  • apply role(gitlab) to gitlab1003 and verify installation 791599
  • copy ssh host keys for ssh-gitlab daemon from gitlab2001 to gitlab1003
  • configure gitlab1003 with profile::gitlab::service_name: 'gitlab-replica.wikimedia.org' 801651
  • configure gitlab1004 as profile::gitlab::active_host not needed on replica
  • apply gitlab-settings to gitlab1003
  • announce downtime some days ahead on ops/releng list? not needed on replica

Scheduled downtime:

  • Announce downtime in #wikimedia-gitlab not needed on replica
  • pause all GitLab Runners not needed on replica
  • stop puppet on gitlab2001 with sudo disable-puppet "Full backup - T307142"
  • stop GitLab on gitlab2001 with gitlab-ctl stop
  • create full backup on gitlab1001/gitlab2001 with /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE="true" GITLAB_BACKUP_MAX_CONCURRENCY="4" GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY="1" && ls -t "/mnt/gitlab-backup"/*gitlab_backup.tar | head -n1 | xargs -i cp {} "/mnt/gitlab-backup"/latest/latest-data.tar
  • sync backup, on gitlab1001 run /usr/bin/rsync -avp /mnt/gitlab-backup/latest/ rsync://gitlab1003.wikimedia.org/data-backup
  • trigger restore on gitlab1003 run /srv/gitlab-backup/gitlab-restore.sh
  • overwrite home_page_url. on gitlab1003 run echo "ApplicationSetting.last.update(home_page_url: 'https://gitlab-replica.wikimedia.org/explore')" | /usr/bin/gitlab-rails console
  • Point DNS entry for gitlab-replica.wikimedia.org to gitlab1003 801652 and run authdns-update
  • verify installation
  • enable puppet on gitlab2001 with sudo enable-puppet "Full backup - T307142"
  • unpause all GitLab Runners not needed on replica
  • announce end of downtime not needed on replica
gerritbot added a comment.May 31 2022, 2:55 PM

Change 801651 merged by Jelto:

[operations/puppet@production] gitlab: make gitlab1003 new replica

https://gerrit.wikimedia.org/r/801651

gerritbot added a comment.May 31 2022, 4:04 PM

Change 801652 merged by Jelto:

[operations/dns@master] wikimedia.org: make gitlab1003 the new gitlab-replica

https://gerrit.wikimedia.org/r/801652

TheresNoTime added a subscriber: TheresNoTime.May 31 2022, 4:21 PM
Jelto added a subscriber: brennen.EditedMay 31 2022, 4:32 PM

Checklist for gitlab migation from gitlab1001 to gitlab1004:

Preparations before downtime:

  • register second service IPs for gitlab1004
  • validate puppet code and GitLab configuration with a physical replica on gitlab1004 (also bullseye)
  • move gitlab.wikimedia.org entry from netbox to dns repo 800709
  • apply role(gitlab) to gitlab1004 and verify installation 800728
  • copy ssh host keys for ssh-gitlab daemon from gitlab1001 to gitlab1004
  • apply gitlab-settings to gitlab1004
  • lower TTL for gitlab.wikimedia.org? 802090
  • announce downtime some days ahead on ops/releng list? @brennen is doing that

Scheduled downtime:

  • Announce downtime in #wikimedia-gitlab
  • pause all GitLab Runners
  • stop puppet on gitlab1001 with sudo disable-puppet "Full backup - T307142"
  • stop write access on nginx and ssh-gitlab on gitlab1001 with gitlab-ctl stop nginx and systemctl stop ssh-gitlab
  • create full backup on gitlab1001 with /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE="true" GITLAB_BACKUP_MAX_CONCURRENCY="4" GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY="1" && ls -t "/mnt/gitlab-backup"/*gitlab_backup.tar | head -n1 | xargs -i cp {} "/mnt/gitlab-backup"/latest/latest-data.tar
  • sync backup, on gitlab1001 run /usr/bin/rsync -avp /mnt/gitlab-backup/latest/ rsync://gitlab1004.wikimedia.org/data-backup
  • configure gitlab1004 with profile::gitlab::service_name: 'gitlab.wikimedia.org' 802150
  • configure gitlab1004 as profile::gitlab::active_host 802150
  • trigger restore on gitlab1004 run /srv/gitlab-backup/gitlab-restore.sh
  • overwrite home_page_url. on gitlab1004 run echo "ApplicationSetting.last.update(home_page_url: 'https://gitlab.wikimedia.org/explore')" | /usr/bin/gitlab-rails console
  • Point DNS entry for gitlab.wikimedia.org to gitlab1004 802473 and run authdns-update
  • verify installation
  • run puppet on gitlab1004
  • enable puppet on gitlab1001 with sudo enable-puppet "Full backup - T307142"
  • unpause all GitLab Runners
  • announce end of downtime
gerritbot added a comment.Jun 1 2022, 10:27 AM

Change 802090 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/dns@master] wikimedia.org: reduce TTL for gitlab A and AAAA to 5m

https://gerrit.wikimedia.org/r/802090

gerritbot added a comment.Jun 1 2022, 2:42 PM

Change 802150 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: make gitlab1004 new production instance

https://gerrit.wikimedia.org/r/802150

Jelto updated the task description. (Show Details)Jun 1 2022, 2:53 PM
gerritbot added a comment.Jun 2 2022, 9:30 AM

Change 802090 merged by Jelto:

[operations/dns@master] wikimedia.org: reduce TTL for gitlab A and AAAA to 5m

https://gerrit.wikimedia.org/r/802090

gerritbot added a comment.Jun 2 2022, 10:08 AM

Change 802473 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/dns@master] wikimedia.org: make gitlab1004 the new gitlab production host

https://gerrit.wikimedia.org/r/802473

Stashbot added a comment.Jun 2 2022, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-02T15:06:14Z] <jelto> start migration to gitlab1004 - T307142

gerritbot added a comment.Jun 2 2022, 3:35 PM

Change 802150 merged by Jelto:

[operations/puppet@production] gitlab: make gitlab1004 new production instance

https://gerrit.wikimedia.org/r/802150

gerritbot added a comment.Jun 2 2022, 3:53 PM

Change 802473 merged by Jelto:

[operations/dns@master] wikimedia.org: make gitlab1004 the new gitlab production host

https://gerrit.wikimedia.org/r/802473

Jelto added a comment.Jun 2 2022, 4:49 PM

Migration of production GitLab from gitlab1001 to gitlab1004 was successful. Downtime was around 65 minutes.

Backups were configured for the new (old) path /srv/gitlab-backup/ and a manual backup run on gitlab1004 was successful.

I'll check bacula tomorrow for new backups of gitlab1004 and make sure the replicas synced properly.

Jelto added a comment.Jun 3 2022, 8:51 AM
In T307142#7976964, @Jelto wrote:

...
I'll check bacula tomorrow for new backups of gitlab1004 and make sure the replicas synced properly.

Backups for the new production host gitlab1004 look as expected. Backups also appear on Bacula:

Select the Client (1-244): 103
Automatically selected FileSet: gitlab
+---------+-------+----------+----------------+---------------------+----------------+
| JobId   | Level | JobFiles | JobBytes       | StartTime           | VolumeName     |
+---------+-------+----------+----------------+---------------------+----------------+
| 447,261 | F     |        6 | 14,754,287,536 | 2022-06-03 04:58:17 | production0081 |
+---------+-------+----------+----------------+---------------------+----------------+

Restore from Bacula to gitlab1004 worked and the files are similar to todays backup.

Replica restore also worked. One rsync job failed on the old production host gitlab1001. I'll take a closer look. I assume that's a timer/resource which is not managed by puppet anymore.

Jelto updated the task description. (Show Details)Jun 3 2022, 8:51 AM
Jelto added a comment.Jun 3 2022, 3:12 PM
In T307142#7978386, @Jelto wrote:

...
One rsync job failed on the old production host gitlab1001. I'll take a closer look. I assume that's a timer/resource which is not managed by puppet anymore.

Two rsync jobs were not removed by puppet when switching from active hosts to replica. This jobs failed because gitlab1001, now a replica is not allowed to write to gitlab1004.
I removed them manually. I'll try to implement a proper fix for that by using the correct combination of ensure flags in puppet (I guess in gitlab::rsync).

gerritbot added a comment.Jun 3 2022, 7:07 PM

Change 802821 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] Bug: T307142 Change-Id: I13b8b25f66cf7c384ce464bbbeb9b7a3a7dc3861

https://gerrit.wikimedia.org/r/802821

gerritbot added a comment.Jun 3 2022, 7:11 PM

Change 802821 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] logtash: replace gitlab1001 with gitlab1004 in tests

https://gerrit.wikimedia.org/r/802821

gerritbot added a comment.Jun 3 2022, 7:14 PM

Change 802822 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gitlab/acme_chief: remove gitlab1001 from list of (passive) hosts

https://gerrit.wikimedia.org/r/802822

gerritbot added a comment.Jun 3 2022, 7:17 PM

Change 802824 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] DHCP: remove gitlab1001

https://gerrit.wikimedia.org/r/802824

gerritbot added a comment.Jun 3 2022, 7:19 PM

Change 802846 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: remove gitlab1001, adjust gitlab machine descriptions

https://gerrit.wikimedia.org/r/802846

gerritbot added a comment.Jun 3 2022, 7:54 PM

Change 802821 merged by Dzahn:

[operations/puppet@production] logtash: replace gitlab1001 with gitlab1004 in tests

https://gerrit.wikimedia.org/r/802821

LSobanski added a subscriber: LSobanski.Jun 7 2022, 2:29 PM
Jelto mentioned this in T296713: Document and test failover for GitLab and GitLab Replica.Jun 8 2022, 2:33 PM
Jelto mentioned this in T310265: Reduce usage of public IPv4 addresses on GitLab hosts.Jun 9 2022, 11:39 AM
gerritbot added a comment.Jun 16 2022, 6:20 PM

Change 806273 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] install_server: remove gitlab-runner1001

https://gerrit.wikimedia.org/r/806273

gerritbot added a comment.Jun 16 2022, 6:21 PM

Change 806273 merged by Dzahn:

[operations/puppet@production] install_server: remove gitlab-runner1001

https://gerrit.wikimedia.org/r/806273

gerritbot added a comment.Jun 16 2022, 6:38 PM

Change 806276 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] install_server: remove gitlab-runner 2001

https://gerrit.wikimedia.org/r/806276

gerritbot added a comment.Jun 16 2022, 6:45 PM

Change 806276 merged by AOkoth:

[operations/puppet@production] install_server: remove gitlab-runner 2001

https://gerrit.wikimedia.org/r/806276

Dzahn updated the task description. (Show Details)Jun 16 2022, 6:51 PM
ops-monitoring-bot added a comment.Jun 16 2022, 6:53 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: gitlab-runner1001.eqiad.wmnet

  • gitlab-runner1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
gerritbot added a comment.Jun 16 2022, 6:57 PM

Change 806279 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] site: remove old gitlab runners

https://gerrit.wikimedia.org/r/806279

ops-monitoring-bot added a comment.Jun 16 2022, 7:00 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: gitlab-runner1001.eqiad.wmnet

  • gitlab-runner1001.eqiad.wmnet (FAIL)
    • Host steps raised exception:

ERROR: some step on some host failed, check the bolded items above

Dzahn mentioned this in T310831: DNS cookbook failed syncing with netbox - 403 from netbox1002.Jun 16 2022, 7:08 PM
ops-monitoring-bot added a comment.Jun 16 2022, 7:39 PM

cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: gitlab-runner2001.codfw.wmnet

  • gitlab-runner2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
gerritbot added a comment.Jun 17 2022, 6:03 PM

Change 806279 merged by AOkoth:

[operations/puppet@production] site: remove old gitlab runners

https://gerrit.wikimedia.org/r/806279

Jelto added a comment.Jun 20 2022, 9:11 AM

@Arnoldokoth @Dzahn what's missing to also check

  • decommission gitlab-runner1001.eqiad.wmnet
  • decommission gitlab-runner2001.codfw.wmnet

in the task description?

gerritbot added a comment.Jun 20 2022, 9:55 AM

Change 806862 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] DHCP: remove gitlab2001

https://gerrit.wikimedia.org/r/806862

gerritbot added a comment.Jun 20 2022, 9:55 AM

Change 806863 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab/acme_chief: remove gitlab2001 from list of (passive) hosts

https://gerrit.wikimedia.org/r/806863

gerritbot added a comment.Jun 20 2022, 9:55 AM

Change 806864 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] site: remove gitlab2001

https://gerrit.wikimedia.org/r/806864

Arnoldokoth updated the task description. (Show Details)Jun 21 2022, 3:15 PM
gerritbot added a comment.Jun 23 2022, 8:52 PM

Change 806863 merged by AOkoth:

[operations/puppet@production] gitlab/acme_chief: remove gitlab2001 from list of (passive) hosts

https://gerrit.wikimedia.org/r/806863

gerritbot added a comment.Jun 23 2022, 8:57 PM

Change 806862 merged by AOkoth:

[operations/puppet@production] DHCP: remove gitlab2001

https://gerrit.wikimedia.org/r/806862

ops-monitoring-bot added a comment.Jun 27 2022, 6:20 PM

cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: gitlab2001.wikimedia.org

  • gitlab2001.wikimedia.org (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Host steps raised exception: Cannot find cluster row_D (expected ('ganeti01.svc.eqiad.wmnet', 'ganeti01.svc.codfw.wmnet', 'ganeti01.svc.esams.wmnet', 'ganeti01.svc.ulsfo.wmnet', 'ganeti01.svc.eqsin.wmnet', 'ganeti-test01.svc.codfw.wmnet', 'ganeti01.svc.drmrs.wmnet', 'ganeti02.svc.drmrs.wmnet')).

ERROR: some step on some host failed, check the bolded items above

Jelto mentioned this in T295793: Support services VIPs with not marked as VIP in Netbox.Jul 5 2022, 7:23 AM
Dzahn added a comment.Jul 5 2022, 6:15 PM

gitlab2001 has been removed from the acme_chief yaml, that allowed it to request certs but it's still up and has the puppet role applied.

This means puppet is broken there with "Acme_chief::Cert[gitlab]/File[/etc/acmecerts/gitlab]: Could not evaluate: Could not retrieve file metadata for puppet://acmechief1001.eqiad.wmnet/acmedata/gitlab: Error 403 on SERVER: access denied"

gerritbot added a comment.Jul 5 2022, 6:19 PM

Change 811351 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gitlab: switch gitlab2001 back to "insetup" role

https://gerrit.wikimedia.org/r/811351

gerritbot added a comment.Jul 5 2022, 6:21 PM

Change 811351 merged by Dzahn:

[operations/puppet@production] gitlab: switch gitlab2001 back to "insetup" role

https://gerrit.wikimedia.org/r/811351

Dzahn added a comment.Jul 5 2022, 6:31 PM
In T307142#7967245, @Jelto wrote:

https://gitlab-replica.wikimedia.org/ will be migrated from gitlab2001 (the old ganeti VM) to gitlab1003 (the new physical machine).

Just noticed in netbox that is still 208.80.153.105 (which is now gitlab-replica-old). I removed the gitlab prod role from gitlab2001 though and confirmed 208.80.153.105 is indeed on gitlab1003 and still up and running.

ops-monitoring-bot added a comment.Jul 5 2022, 6:39 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: gitlab2001.codfw.wmnet

  • gitlab2001.codfw.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • Failed to shutdown VM, manually run gnt-instance remove on the Ganeti master for the codfw cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • Failed to remove VM, manually run gnt-instance remove on the Ganeti master for the codfw cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Jul 5 2022, 6:52 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: gitlab2001.wikimedia.org

  • gitlab2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
Dzahn mentioned this in T311446: errors decom'ing VMs (was: pending diff in sre.dns.netbox cookbook).Jul 5 2022, 6:54 PM
Dzahn added a subtask: T311446: errors decom'ing VMs (was: pending diff in sre.dns.netbox cookbook).
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jul 5 2022, 7:01 PM

Change 811362 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/hiera: remove gitlab2001 after decom

https://gerrit.wikimedia.org/r/811362

Stashbot added a comment.Jul 6 2022, 1:21 AM

Mentioned in SAL (#wikimedia-operations) [2022-07-06T01:21:25Z] <mutante> gitlab1004 rm /lib/systemd/system/rsync-data-backup-gitlab2001.wikimedia.org.* ; systemctl reset-failed (T274463, T307142) - fix icinga alert after gitlab2001 was decom'ed, we didn't have puppet remove the timer/service

gerritbot added a comment.Jul 6 2022, 9:20 AM

Change 811362 merged by Jelto:

[operations/puppet@production] site/hiera: remove gitlab2001 after decom

https://gerrit.wikimedia.org/r/811362

gerritbot added a comment.Jul 6 2022, 9:34 AM

Change 811674 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/dns@master] wikimedia.org: remove gitlab-replica-old.wikimedia.org

https://gerrit.wikimedia.org/r/811674

gerritbot added a comment.Jul 6 2022, 11:22 AM

Change 811674 merged by Jelto:

[operations/dns@master] wikimedia.org: remove gitlab-replica-old.wikimedia.org

https://gerrit.wikimedia.org/r/811674

gerritbot added a comment.Jul 6 2022, 11:45 AM

Change 806864 abandoned by Jelto:

[operations/puppet@production] site: remove gitlab2001

Reason:

duplicate with I7f409741d59e0a951f6f24470cb5d309bb7d2e6d

https://gerrit.wikimedia.org/r/806864

gerritbot added a comment.Jul 6 2022, 10:50 PM

Change 802822 merged by Dzahn:

[operations/puppet@production] gitlab/acme_chief: remove gitlab1001 from list of (passive) hosts

https://gerrit.wikimedia.org/r/802822

Stashbot added a comment.Jul 6 2022, 11:00 PM

Mentioned in SAL (#wikimedia-operations) [2022-07-06T23:00:01Z] <mutante> gitlab1004 - rm /lib/systemd/system/rsync-config-backup-gitlab1001* T307142

gerritbot added a comment.Jul 6 2022, 11:07 PM

Change 811782 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/gitlab: remove gitlab1001, update comments

https://gerrit.wikimedia.org/r/811782

gerritbot added a comment.Jul 6 2022, 11:10 PM

Change 802824 merged by Dzahn:

[operations/puppet@production] DHCP: remove gitlab1001

https://gerrit.wikimedia.org/r/802824

gerritbot added a comment.Jul 6 2022, 11:18 PM

Change 802846 abandoned by Dzahn:

[operations/puppet@production] site: remove gitlab1001, adjust gitlab machine descriptions

Reason:

duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/+/811782

https://gerrit.wikimedia.org/r/802846

ops-monitoring-bot added a comment.Jul 7 2022, 12:25 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: gitlab1001.wikimedia.org

  • gitlab1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
Dzahn updated the task description. (Show Details)Jul 7 2022, 12:30 AM
gerritbot added a comment.Jul 7 2022, 1:31 AM

Change 811782 merged by Dzahn:

[operations/puppet@production] site/gitlab: remove gitlab1001, update comments

https://gerrit.wikimedia.org/r/811782

gerritbot added a comment.Jul 7 2022, 1:33 AM

Change 811797 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gitlab: remove gitlab1001 from Hiera

https://gerrit.wikimedia.org/r/811797

Dzahn updated the task description. (Show Details)Jul 7 2022, 1:33 AM
gerritbot added a comment.Jul 7 2022, 1:35 AM

Change 811797 merged by Dzahn:

[operations/puppet@production] gitlab: remove gitlab1001 from Hiera

https://gerrit.wikimedia.org/r/811797

gerritbot added a comment.Jul 7 2022, 8:57 AM

Change 811912 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/dns@master] wikimedia.org: remove gitlab-old.wikimedia.org

https://gerrit.wikimedia.org/r/811912

Dzahn updated the task description. (Show Details)Jul 7 2022, 8:10 PM

old VMs completely gone now. all decom boxes checked.

Stashbot added a comment.Jul 8 2022, 1:12 AM

Mentioned in SAL (#wikimedia-operations) [2022-07-08T01:12:43Z] <mutante> gitlab1004 - _still_ icinga alerts about rsync to decom'ed host. 'systemctl daemon-reload' to teach it about deleted units, then systemctl reset failed ..then RECOVERY T307142

gerritbot added a comment.Jul 8 2022, 9:31 AM

Change 811912 merged by Jelto:

[operations/dns@master] wikimedia.org: remove gitlab-old.wikimedia.org

https://gerrit.wikimedia.org/r/811912

gerritbot added a comment.Jul 25 2022, 6:59 PM

Change 816835 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] gitlab: add reserved service IP 208.80.154.8, point to replica-new

https://gerrit.wikimedia.org/r/816835

gerritbot added a comment.Jul 29 2022, 4:56 PM

Change 816835 merged by Dzahn:

[operations/dns@master] gitlab: add reserved service IP 208.80.153.8, point to replica-new

https://gerrit.wikimedia.org/r/816835

Dzahn added a comment.Jul 29 2022, 4:58 PM

@Arnoldokoth The DNS change for gitlab-replica-new has been merged and deployed on DNS servers now:

[authdns1001:~] $ host gitlab-replica-new.wikimedia.org
gitlab-replica-new.wikimedia.org has address 208.80.153.8
gitlab-replica-new.wikimedia.org has IPv6 address 2620:0:860:1:208:80:153:8
[authdns1001:~] $ host 208.80.153.8
8.153.80.208.in-addr.arpa domain name pointer gitlab-replica-new.wikimedia.org.
[authdns1001:~] $ host 2620:0:860:1:208:80:153:8
8.0.0.0.3.5.1.0.0.8.0.0.8.0.2.0.1.0.0.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa domain name pointer gitlab-replica-new.wikimedia.org.
jijiki raised the priority of this task from High to Unbreak Now!.Nov 8 2022, 6:46 PM
jijiki lowered the priority of this task from Unbreak Now! to High.
jijiki edited projects, added serviceops-collab; removed serviceops.
LSobanski lowered the priority of this task from High to Medium.Nov 15 2022, 4:06 PM
LSobanski moved this task from Incoming to Work in Progress on the serviceops-collab board.
Jelto mentioned this in T323201: Evaluate a high available GitLab architecture.Nov 16 2022, 9:50 AM
Jelto closed this task as Resolved.Nov 16 2022, 9:52 AM

All GitLab machines in codfw and eqiad are migrated to physical hosts. I created a follow-up task T323201 to discuss what should happen with the remaining host gitlab2003 and also with the second replica gitlab1003. I'm closing this task.

Jelto updated the task description. (Show Details)Nov 16 2022, 9:53 AM
Dzahn awarded a token.Nov 17 2022, 6:49 PM
Jelto mentioned this in T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).Feb 17 2023, 10:00 AM
Jelto mentioned this in T329931: Switchover gitlab (gitlab1004 -> gitlab2002).Feb 17 2023, 10:04 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL