Page MenuHomePhabricator

Jelto (jwodstrcil)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Jun 7 2021, 7:25 AM (94 w, 2 d)
Availability
Available
LDAP User
Jelto
MediaWiki User
JWodstrcil (WMF) [ Global Accounts ]

Recent Activity

Today

Jelto added a comment to T332869: Phabricator's access log may have some problems in log rotation.

After some troulbleshooting with @eoghan we found out systemd::unmask has set the refreshonly parameter to true in the exec command. Which means

Wed, Mar 29, 10:14 AM · Phabricator, serviceops-collab
Jelto added a comment to T332869: Phabricator's access log may have some problems in log rotation.

With merging https://gerrit.wikimedia.org/r/903693 the aphlict logrotate job was removed from phabricator hosts and stayed on aphlict host.

Wed, Mar 29, 8:53 AM · Phabricator, serviceops-collab
Jelto updated the task description for T330165: eqiad row B switches upgrade.
Wed, Mar 29, 7:29 AM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic

Yesterday

Jelto updated subscribers of T332869: Phabricator's access log may have some problems in log rotation.

thanks to @eoghan for pointing me to a patch which disabled logrotate on phabricator hosts: https://gerrit.wikimedia.org/r/c/operations/puppet/+/902396/8/modules/phabricator/manifests/aphlict.pp

Tue, Mar 28, 3:54 PM · Phabricator, serviceops-collab
Jelto claimed T332869: Phabricator's access log may have some problems in log rotation.

I can confirm log rotate failed 4 days ago. It seems to be stuck (inactive (dead)):

Tue, Mar 28, 2:00 PM · Phabricator, serviceops-collab
Jelto updated the task description for T330165: eqiad row B switches upgrade.
Tue, Mar 28, 1:46 PM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic
Jelto updated the task description for T330165: eqiad row B switches upgrade.
Tue, Mar 28, 1:41 PM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic
Jelto added a comment to T324506: Investigate incremental backups for GitLab.

Thanks @Arnoldokoth for the tests. Yes it seems the "incremental" backup is significantly smaller and faster than a full backup. But it's not fully clear to me what's happening internally and what backup file we need for a restore. Documentation around the incremental backup feature is quite limited. Also the output of backup jobs is very limited (when leaving the CRON=1 parameter).

Tue, Mar 28, 12:44 PM · GitLab (Infrastructure), serviceops-collab
Jelto created T333304: Install additional SSDs on gitlab2003.wikimedia.org (B5).
Tue, Mar 28, 8:39 AM · SRE, ops-codfw, GitLab (Infrastructure), serviceops-collab

Mon, Mar 27

Jelto added a comment to T320730: Define access to external resources for GitLab CI Runners.

I think policy is now that Trusted runners only use internal images; would it be possible therefore to start making an internal copy of registry.gitlab.com/gitlab-org/release-cli, please? It's really useful if wanting to make gitlab releases from CI (which would be handy as and when I want to start making .debs from CI) - e.g. currently being used for https://gitlab.wikimedia.org/repos/data_persistence/wmf-beamer-style releases.

Mon, Mar 27, 1:27 PM · Release-Engineering-Team, GitLab (CI & Job Runners), serviceops-collab
Jelto created T333161: self-build/import registry.gitlab.com/gitlab-org/release-cli image.
Mon, Mar 27, 1:25 PM · GitLab (CI & Job Runners), serviceops-collab
Jelto updated the task description for T330165: eqiad row B switches upgrade.
Mon, Mar 27, 11:34 AM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic
Jelto updated the task description for T331882: eqiad row C switches upgrade.
Mon, Mar 27, 11:24 AM · Patch-For-Review, serviceops-radar, Discovery-Search (Current work), SRE, DBA, cloud-services-team, Traffic, Infrastructure-Foundations, Machine-Learning-Team, Data-Engineering, serviceops-collab, Platform Engineering, SRE Observability

Thu, Mar 16

Dzahn awarded T332114: GitLab restore failed on gitlab1003 a Haypence token.
Thu, Mar 16, 1:00 AM · GitLab (Infrastructure), serviceops-collab

Wed, Mar 15

Jelto added a project to T331295: Add safeguard flag to gitlab-restore.sh script: Sustainability (Incident Followup).
Wed, Mar 15, 12:04 PM · Sustainability (Incident Followup), serviceops-collab, GitLab (Infrastructure)
Jelto closed T332114: GitLab restore failed on gitlab1003 as Resolved.

The restore failed with a deadlock in the postgres DB restore:

Wed, Mar 15, 9:00 AM · GitLab (Infrastructure), serviceops-collab
Jelto triaged T332114: GitLab restore failed on gitlab1003 as Medium priority.
Wed, Mar 15, 8:13 AM · GitLab (Infrastructure), serviceops-collab
Jelto moved T332114: GitLab restore failed on gitlab1003 from Incoming to Work in Progress on the serviceops-collab board.
Wed, Mar 15, 8:13 AM · GitLab (Infrastructure), serviceops-collab
Jelto created T332114: GitLab restore failed on gitlab1003.
Wed, Mar 15, 8:13 AM · GitLab (Infrastructure), serviceops-collab

Tue, Mar 14

Jelto awarded T322369: create aphlict2001 (Phabricator realtime notifications codfw) a Yellow Medal token.
Tue, Mar 14, 10:16 AM · Phabricator, serviceops-collab
Jelto awarded T327975: create blackbox::http monitoring for releases.wikimedia.org a Yellow Medal token.
Tue, Mar 14, 8:28 AM · serviceops-collab

Mon, Mar 13

Jelto reopened T327975: create blackbox::http monitoring for releases.wikimedia.org, a subtask of T329587: http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets), as Open.
Mon, Mar 13, 1:11 PM · Patch-For-Review, serviceops-collab
Jelto reopened T327975: create blackbox::http monitoring for releases.wikimedia.org as "Open".

blackbox checks for releases fail since it was added to prometheus: https://logstash.wikimedia.org/goto/a0d856fad268c5301031fb6e47c56b9b and https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes%2Fcustom&var-module=All&orgId=1&viewPanel=3. A puppet issue on prometheus prevented the blackbox check from being configured properly before.

Mon, Mar 13, 1:11 PM · serviceops-collab
Jelto added a comment to T329679: Set up mirror of the docker hub registry for gitlab-runners.

A docker registry container is running on runner-1029 in WMCS now. A quick test with enabling the pull-through cache seemed to work. With enabled pull-through cache and a docker pull, the image data ends up in /var/lib/docker-registry.

Mon, Mar 13, 10:40 AM · Patch-For-Review, serviceops-radar, serviceops-collab, Release-Engineering-Team (GitLab V: Event Horizon 🌄), GitLab (CI & Job Runners)

Wed, Mar 8

Jelto closed T331295: Add safeguard flag to gitlab-restore.sh script as Resolved.

The restore script has a safeguard implemented on production (see change above):

Wed, Mar 8, 10:06 AM · Sustainability (Incident Followup), serviceops-collab, GitLab (Infrastructure)

Tue, Mar 7

Jelto closed T331296: Create cumin host in devtools project as Declined.

I deleted the new host cumin-master-1001.devtools.eqiad1.wikimedia.cloud.

Tue, Mar 7, 4:51 PM · serviceops-collab, GitLab (Infrastructure)
Jelto added a comment to T331296: Create cumin host in devtools project.

@Jelto the TL;DR is that there is no real support for Spicerack and cookbooks in WMCS for the use case you're looking for simply because there are too many differences between production and WMCS and there is no project in WMCS that replicates all the services required by Spicerack. So I don't think that with the current state of things this approach is really feasible / will actually work.

Tue, Mar 7, 3:49 PM · serviceops-collab, GitLab (Infrastructure)
Jelto moved T331295: Add safeguard flag to gitlab-restore.sh script from Incoming to Work in Progress on the serviceops-collab board.
Tue, Mar 7, 3:30 PM · Sustainability (Incident Followup), serviceops-collab, GitLab (Infrastructure)
Jelto claimed T331295: Add safeguard flag to gitlab-restore.sh script.
Tue, Mar 7, 3:30 PM · Sustainability (Incident Followup), serviceops-collab, GitLab (Infrastructure)
Jelto added a comment to T330172: Define future design of GitLab backups.

The above estimates consider just the current GitLab usage. At some point we may migrate all projects from gerrit to GitLab. The sizes on gerrit1001 are:

Tue, Mar 7, 2:40 PM · serviceops-collab, GitLab (Infrastructure)
Jelto updated subscribers of T331296: Create cumin host in devtools project.

Adding @Volans to get some feedback about running cookbooks in WMCS. I'm not sure if creating a dedicated cumin master helps us running cookbooks in WMCS.

Tue, Mar 7, 1:26 PM · serviceops-collab, GitLab (Infrastructure)

Mon, Mar 6

Jelto updated the task description for T329073: eqiad row A switches upgrade.
Mon, Mar 6, 3:36 PM · Patch-For-Review, Discovery-Search (Current work), Shared-Data-Infrastructure, Data-Engineering-Planning, DBA, SRE, Platform Engineering, Infrastructure-Foundations, Traffic, serviceops, Machine-Learning-Team, cloud-services-team, Data-Persistence, SRE Observability, serviceops-collab
Jelto added a comment to T331296: Create cumin host in devtools project.

I created cumin-master-1001 in devtools project and configured the host according to https://wikitech.wikimedia.org/wiki/Help:Cumin_master.

Mon, Mar 6, 3:23 PM · serviceops-collab, GitLab (Infrastructure)
Jelto triaged T331296: Create cumin host in devtools project as Medium priority.
Mon, Mar 6, 2:17 PM · serviceops-collab, GitLab (Infrastructure)
Jelto moved T331296: Create cumin host in devtools project from Incoming to Work in Progress on the serviceops-collab board.
Mon, Mar 6, 2:17 PM · serviceops-collab, GitLab (Infrastructure)
Jelto created T331296: Create cumin host in devtools project.
Mon, Mar 6, 2:17 PM · serviceops-collab, GitLab (Infrastructure)
Jelto created T331295: Add safeguard flag to gitlab-restore.sh script.
Mon, Mar 6, 2:12 PM · Sustainability (Incident Followup), serviceops-collab, GitLab (Infrastructure)

Thu, Mar 2

Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

The rsync jobs between production host and replica are only created but not removed, when the list of replicas change. On the former production instance gitlab1004 the jobs rsync-config-backup-gitlab2002.wikimedia.org.timer and rsync-data-backup-gitlab2002.wikimedia.org.timer are still present.

https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/gitlab/manifests/rsync.pp#40

The jobs failed because rsync server is disabled automatically on the new production instance. However we should find a way to manag the rsync jobs properly. Removing the jobs by hand after each failover sounds error prone.

Thu, Mar 2, 3:04 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto added a comment to T329679: Set up mirror of the docker hub registry for gitlab-runners.

One of the WMCS Shared Runners runner-29 has a dedicated disk for storing the mirrored dockerhub images now:

Thu, Mar 2, 8:07 AM · Patch-For-Review, serviceops-radar, serviceops-collab, Release-Engineering-Team (GitLab V: Event Horizon 🌄), GitLab (CI & Job Runners)

Wed, Mar 1

Jelto added a comment to T330172: Define future design of GitLab backups.

I've done some more research on the above ideas:

Wed, Mar 1, 5:34 PM · serviceops-collab, GitLab (Infrastructure)
Jelto added a comment to T323515: Create VRTS Replica VM in codfw.

^ the alert is firing again. I guess the downtime expired?

Wed, Mar 1, 9:55 AM · vrts, Znuny, serviceops-collab

Tue, Feb 28

Jelto added a comment to T330744: gitlab backup timer failing.

Time/jobs for gitlab2002 were removed on gitlab1004:

Tue, Feb 28, 1:13 PM · serviceops-collab, GitLab
Jelto added a project to T330744: gitlab backup timer failing: serviceops-collab.

thanks for keeping an eye on that!

Tue, Feb 28, 12:01 PM · serviceops-collab, GitLab
Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

The rsync jobs between production host and replica are only created but not removed, when the list of replicas change. On the former production instance gitlab1004 the jobs rsync-config-backup-gitlab2002.wikimedia.org.timer and rsync-data-backup-gitlab2002.wikimedia.org.timer are still present.

Tue, Feb 28, 11:19 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto closed T330717: ProbeDown as Resolved.

After merging the above change the restore timer is gone form gitlab2002. So we should not see a ProbeDown alert again due to restores on the production instance.

Tue, Feb 28, 10:38 AM · GitLab (Infrastructure), Patch-For-Review, serviceops-collab
Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

I started a incident report in 2023-02-28_GitLab_data_loss

Tue, Feb 28, 9:01 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

As mentioned in T330717, the new production host gitlab2002 still had the restore enabled and executed a restore this night.

Tue, Feb 28, 8:33 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto added a project to T330717: ProbeDown: GitLab (Infrastructure).
Tue, Feb 28, 8:27 AM · GitLab (Infrastructure), Patch-For-Review, serviceops-collab
Jelto added a comment to T330717: ProbeDown.

So gitlab2002 was down for around 20 minutes while doing the restore. The above change should disable restore on the production host.

Tue, Feb 28, 8:27 AM · GitLab (Infrastructure), Patch-For-Review, serviceops-collab
Jelto claimed T330717: ProbeDown.

gitlab2002 was switched from replica to a production instance yesterday in T329931.

Tue, Feb 28, 7:53 AM · GitLab (Infrastructure), Patch-For-Review, serviceops-collab

Mon, Feb 27

thcipriani awarded T329931: Switchover gitlab (gitlab1004 -> gitlab2002) a Yellow Medal token.
Mon, Feb 27, 6:51 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto moved T329571: cleanup mariadb datadir situation on cloud VPS hosts using simplelamp role from Incoming to Backlog on the serviceops-collab board.
Mon, Feb 27, 5:01 PM · serviceops-collab, Patch-For-Review
Jelto triaged T329571: cleanup mariadb datadir situation on cloud VPS hosts using simplelamp role as Medium priority.
Mon, Feb 27, 5:01 PM · serviceops-collab, Patch-For-Review
Jelto assigned T329571: cleanup mariadb datadir situation on cloud VPS hosts using simplelamp role to Dzahn.
Mon, Feb 27, 5:00 PM · serviceops-collab, Patch-For-Review
Jelto moved T330282: gitlab_default_can_create_group setting deprecation from Incoming to Backlog on the serviceops-collab board.
Mon, Feb 27, 4:59 PM · Patch-For-Review, GitLab (Infrastructure), serviceops-collab
Jelto assigned T330282: gitlab_default_can_create_group setting deprecation to eoghan.
Mon, Feb 27, 4:59 PM · Patch-For-Review, GitLab (Infrastructure), serviceops-collab
Jelto moved T329444: Issues with Gerrit test instance from Incoming to Backlog on the serviceops-collab board.
Mon, Feb 27, 4:58 PM · Patch-For-Review, serviceops-collab, Gerrit
Jelto assigned T329444: Issues with Gerrit test instance to Dzahn.

@Dzahn can you follow up on the task status?

Mon, Feb 27, 4:58 PM · Patch-For-Review, serviceops-collab, Gerrit
Jelto moved T329679: Set up mirror of the docker hub registry for gitlab-runners from Incoming to Consultation on the serviceops-collab board.
Mon, Feb 27, 4:56 PM · Patch-For-Review, serviceops-radar, serviceops-collab, Release-Engineering-Team (GitLab V: Event Horizon 🌄), GitLab (CI & Job Runners)
Jelto moved T330312: Address Gerrit WMCS instance authenticating against LDAP (breaching WMCS policy) from Incoming to Consultation on the serviceops-collab board.
Mon, Feb 27, 4:55 PM · SecTeam-Processed, Release-Engineering-Team (Yak Shaving 🐃🪒), serviceops-collab, Security, Security-Team
Jelto moved T329073: eqiad row A switches upgrade from Incoming to Consultation on the serviceops-collab board.
Mon, Feb 27, 4:54 PM · Patch-For-Review, Discovery-Search (Current work), Shared-Data-Infrastructure, Data-Engineering-Planning, DBA, SRE, Platform Engineering, Infrastructure-Foundations, Traffic, serviceops, Machine-Learning-Team, cloud-services-team, Data-Persistence, SRE Observability, serviceops-collab
Jelto assigned T330129: Puppet template for /etc/clamav/clamd.conf needs to be updated to Arnoldokoth.
Mon, Feb 27, 4:54 PM · serviceops-collab, Znuny, SRE
Jelto moved T330129: Puppet template for /etc/clamav/clamd.conf needs to be updated from Incoming to Work in Progress on the serviceops-collab board.
Mon, Feb 27, 4:52 PM · serviceops-collab, Znuny, SRE
Jelto moved T330165: eqiad row B switches upgrade from Incoming to Consultation on the serviceops-collab board.
Mon, Feb 27, 4:51 PM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic
Jelto moved T330172: Define future design of GitLab backups from Backlog to Work in Progress on the serviceops-collab board.
Mon, Feb 27, 4:42 PM · serviceops-collab, GitLab (Infrastructure)
Jelto moved T330090: Switchover static miscweb services to codfw from Backlog to Work in Progress on the serviceops-collab board.
Mon, Feb 27, 4:35 PM · Patch-For-Review, Datacenter-Switchover, serviceops-collab

Feb 27 2023

Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

A "Backup freshness" alert with summary:"No backups: 1 (gitlab2002), Fresh: 116 jobs"" triggered. This should resolve on the next bacula run this night. Bacula was disabled for the old instance gitlab1004 and enabled for gitlab2002.

Feb 27 2023, 12:59 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
valerio.bozzolan awarded T329931: Switchover gitlab (gitlab1004 -> gitlab2002) a Like token.
Feb 27 2023, 12:22 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto awarded T329931: Switchover gitlab (gitlab1004 -> gitlab2002) a Like token.
Feb 27 2023, 12:03 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

Maintenance finished, GitLab is back again. If you face any issues feel free to post it here.

Feb 27 2023, 12:03 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 12:02 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 11:58 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 11:51 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 11:36 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 11:25 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 11:03 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 10:19 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 10:18 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 10:07 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto moved T329931: Switchover gitlab (gitlab1004 -> gitlab2002) from Backlog to Work in Progress on the serviceops-collab board.
Feb 27 2023, 9:59 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto assigned T329931: Switchover gitlab (gitlab1004 -> gitlab2002) to eoghan.
Feb 27 2023, 9:59 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 9:35 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 27 2023, 8:24 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover

Feb 23 2023

Jelto moved T328283: Cleanup and attach volumes for gitlab-runners WMCS project from Backlog to Work in Progress on the serviceops-collab board.
Feb 23 2023, 6:41 PM · Release-Engineering-Team (Radar), GitLab (CI & Job Runners), serviceops-collab
Jelto claimed T328283: Cleanup and attach volumes for gitlab-runners WMCS project.

I deleted all unused volumes in the gitlab-runners project. This freed some storage/volume quotas in the project.

Feb 23 2023, 6:41 PM · Release-Engineering-Team (Radar), GitLab (CI & Job Runners), serviceops-collab
Jelto added a comment to T329931: Switchover gitlab (gitlab1004 -> gitlab2002).

I scheduled a new broadcast message on GitLab for the upcoming switchover next Monday. The message should appear tomorrow morning until Monday after the maintenance window.

Feb 23 2023, 6:29 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 23 2023, 9:59 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 23 2023, 9:51 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 23 2023, 9:51 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 23 2023, 9:33 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329931: Switchover gitlab (gitlab1004 -> gitlab2002).
Feb 23 2023, 8:55 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto added a comment to T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).

I think T296944#8496308 is related here.

Feb 23 2023, 8:50 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover

Feb 22 2023

Jelto added a comment to T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).

After switching gitlab2002 and gitlab1003 I get a SSH key warning, although the hosts use the same host key:

Feb 22 2023, 2:56 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 12:32 PM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 11:37 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 11:36 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 11:24 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 11:16 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 10:48 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover
Jelto updated the task description for T329930: Switchover gitlab-replica (gitlab2002 -> gitlab1003).
Feb 22 2023, 10:45 AM · GitLab (Infrastructure), serviceops-collab, Datacenter-Switchover