Page MenuHomePhabricator

Jelto (jwodstrcil)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Jun 7 2021, 7:25 AM (158 w, 3 d)
Availability
Available
LDAP User
Jelto
MediaWiki User
JWodstrcil (WMF) [ Global Accounts ]

Recent Activity

Tue, Jun 18

Jelto added a comment to T366786: GitLabRunnerTrustedConfigMissing.

I think all of the options above can be used. I updated the frequency of the scheduled pipeline from every 30 minutes to every 15 minutes. Also I added a [DO NOT EDIT] description to the Trusted Runners.

Tue, Jun 18, 1:55 PM · Patch-For-Review, collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Tue, Jun 18, 9:03 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto added a project to T364367: Create dedicated UIs for wdqs graph split endpoints: collaboration-services.

Hello! We are transitioning from the legacy Miscweb setup to Kubernetes. Most services have already been migrated (see main tracking task T300171). However, the migration for the query service is still pending (T350793).

Tue, Jun 18, 8:13 AM · collaboration-services, Data-Platform-SRE (2024.06.17 - 2024.07.07), Patch-For-Review, Discovery-Search, Wikidata-Query-Service, Wikidata
Jelto moved T364367: Create dedicated UIs for wdqs graph split endpoints from Incoming to Consultation on the collaboration-services board.
Tue, Jun 18, 8:12 AM · collaboration-services, Data-Platform-SRE (2024.06.17 - 2024.07.07), Patch-For-Review, Discovery-Search, Wikidata-Query-Service, Wikidata

Mon, Jun 17

Jelto raised the priority of T362432: Upgrade etherpad.wikimedia.org to v2.1.0 from Low to Medium.

As discussed in T367544, the VM used for building the Etherpad Debian package will be deleted soon. Therefore, it might be sensible to attempt one last version upgrade. This would give us additional time to evaluate new systems for building and deploying Etherpad (like Kubernetes, wmf-debci or another bookworm build vm).

Mon, Jun 17, 12:40 PM · collaboration-services
Jelto added a comment to T367544: Cloud VPS "packaging" project Buster deprecation.

It might be possible to schedule another Etherpad upgrade (T362432) before the packager02.packaging.eqiad1.wikimedia.cloud host is deleted.

Mon, Jun 17, 12:29 PM · collaboration-services, Cloud-VPS (Debian Buster Deprecation)
Jelto added a comment to T367544: Cloud VPS "packaging" project Buster deprecation.

According to the etherpad upgrade docs this host is used to build the etherpad Debian package. I also used the host in the past to build the etherpad package. The dedicated host is used because "etherpad builds fetches npm modules during the build time".

Mon, Jun 17, 9:53 AM · collaboration-services, Cloud-VPS (Debian Buster Deprecation)
Jelto updated the task description for T367717: Import gitlab-runner package for bookworm on apt hosts.
Mon, Jun 17, 9:43 AM · GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Mon, Jun 17, 9:36 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto created T367717: Import gitlab-runner package for bookworm on apt hosts.
Mon, Jun 17, 9:35 AM · GitLab (Infrastructure), collaboration-services
Jelto added a comment to T365675: Upgrade GitLab to major version 17.

I updated the test instance and test gitlab-runners to GitLab version 17. Everything looks good and the migration was successful. @brennen we can test the account approval bot on the test instance now.

Mon, Jun 17, 9:23 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Mon, Jun 17, 9:19 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services

Fri, Jun 14

Jelto added a comment to T365675: Upgrade GitLab to major version 17.

I'm going to upgrade the test instance to 17.0 next Monday June 17th.

Fri, Jun 14, 1:55 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Fri, Jun 14, 1:52 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto created T367501: GitLab sre.gitlab.upgrade cookbook should lock all other backups during runtime.
Fri, Jun 14, 12:15 PM · collaboration-services, GitLab (Infrastructure)
Jelto closed T367341: GitLabCIJobErrors as Resolved.

This should be fixed with the change above. We ignore user-generated script_failures in the alert now.

Fri, Jun 14, 8:44 AM · GitLab (CI & Job Runners), collaboration-services
Jelto added a parent task for T367481: SystemdUnitFailed (gitlab backup-restore): Unknown Object (Task).
Fri, Jun 14, 7:55 AM · collaboration-services
Jelto closed T367481: SystemdUnitFailed (gitlab backup-restore) as Resolved.

Related to ongoing upgrade in T367382. I silenced the alert. This should resolve later today.

Fri, Jun 14, 7:54 AM · collaboration-services
Jelto renamed T367481: SystemdUnitFailed (gitlab backup-restore) from SystemdUnitFailed (backup-restore) to SystemdUnitFailed (gitlab backup-restore).
Fri, Jun 14, 7:54 AM · collaboration-services
Jelto renamed T367481: SystemdUnitFailed (gitlab backup-restore) from SystemdUnitFailed to SystemdUnitFailed (backup-restore).
Fri, Jun 14, 7:54 AM · collaboration-services

Thu, Jun 13

Jelto moved T367341: GitLabCIJobErrors from Incoming to Work in Progress on the collaboration-services board.
Thu, Jun 13, 2:27 PM · GitLab (CI & Job Runners), collaboration-services
Jelto claimed T367341: GitLabCIJobErrors.

This was triggered by failing jobs due to faulty CI jobs/scripts. Jobs came mostly from https://gitlab.wikimedia.org/repos/security/ci-cd-testing-gitlab-ci-security-templates. So this was user generated.

Thu, Jun 13, 2:27 PM · GitLab (CI & Job Runners), collaboration-services
Jelto closed T367384: SystemdUnitFailed (gitlab1003 exporter) as Resolved.

This is resolved after GitLab is back. I'll try to catch the error in a new version of the exporter.

Thu, Jun 13, 12:03 PM · collaboration-services
Jelto closed T367398: SystemdUnitFailed (gitlab1003 exporter) as Resolved.

This is resolved after GitLab is back. I'll try to catch the error in a new version of the exporter.

Thu, Jun 13, 12:03 PM · collaboration-services
Jelto added a parent task for T367398: SystemdUnitFailed (gitlab1003 exporter): Unknown Object (Task).
Thu, Jun 13, 12:02 PM · collaboration-services
Jelto renamed T367398: SystemdUnitFailed (gitlab1003 exporter) from SystemdUnitFailed to SystemdUnitFailed (gitlab1003 exporter).
Thu, Jun 13, 12:02 PM · collaboration-services
Jelto added a parent task for T367384: SystemdUnitFailed (gitlab1003 exporter): Unknown Object (Task).
Thu, Jun 13, 8:45 AM · collaboration-services
Jelto renamed T367384: SystemdUnitFailed (gitlab1003 exporter) from SystemdUnitFailed to SystemdUnitFailed (gitlab1003 exporter).
Thu, Jun 13, 8:45 AM · collaboration-services

Wed, Jun 12

Jelto closed T367021: Create an SSH blackbox test for GitLab as Resolved.

^ another patch was needed to make sure to probe the service IPs and not the default host IPs (similar to the blackbox::http check).

Wed, Jun 12, 10:29 AM · GitLab, collaboration-services
Jelto closed T367273: ProbeDown (gitlab ssh ipv6), a subtask of T367021: Create an SSH blackbox test for GitLab, as Resolved.
Wed, Jun 12, 10:23 AM · GitLab, collaboration-services
Jelto closed T367273: ProbeDown (gitlab ssh ipv6) as Resolved.

resolved after fixing the IPs which should be probed. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1042191

Wed, Jun 12, 10:23 AM · collaboration-services
Jelto added a subtask for T367021: Create an SSH blackbox test for GitLab: T367273: ProbeDown (gitlab ssh ipv6).
Wed, Jun 12, 10:22 AM · GitLab, collaboration-services
Jelto added a parent task for T367273: ProbeDown (gitlab ssh ipv6): T367021: Create an SSH blackbox test for GitLab.
Wed, Jun 12, 10:22 AM · collaboration-services
Jelto renamed T367273: ProbeDown (gitlab ssh ipv6) from ProbeDown to ProbeDown (gitlab ssh ipv6).
Wed, Jun 12, 10:22 AM · collaboration-services

Tue, Jun 11

Jelto committed rLPRId92119741e78: gitlab: add dummy token for profile::gitlab::configure_projects_bot_token.
gitlab: add dummy token for profile::gitlab::configure_projects_bot_token
Tue, Jun 11, 3:34 PM
Jelto added a comment to T367021: Create an SSH blackbox test for GitLab.

This seemed to be a puppet dependency issue and not a SSH issue. I checked the sshd on all instances and IPv6 worked on both replicas but not on production. I restarted the git sshd and after that sshd listens on IPv6:

Tue, Jun 11, 1:08 PM · GitLab, collaboration-services
Jelto renamed T367114: ProbeDown (moscovium) from ProbeDown to ProbeDown (moscovium).
Tue, Jun 11, 7:23 AM · collaboration-services

Mon, Jun 10

Jelto added a comment to T366786: GitLabRunnerTrustedConfigMissing.

After digging through the docs I did not find any way to restrict the edit page for runners in the projects CI menu, for example here in airflow-dags: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/runners/1484/edit. All users with elevated permissions on the project can edit the CI settings for every runner which is assigned to the project.

Mon, Jun 10, 2:12 PM · Patch-For-Review, collaboration-services
Jelto added a comment to T367021: Create an SSH blackbox test for GitLab.

The gitlab-ssh service is listening on IPv4 only. So the above ProbeDown alert fired for IPv6.

Mon, Jun 10, 11:34 AM · GitLab, collaboration-services
Jelto closed T367038: ProbeDown (gitlab ssh ipv6), a subtask of T367021: Create an SSH blackbox test for GitLab, as Resolved.
Mon, Jun 10, 11:32 AM · GitLab, collaboration-services
Jelto closed T367038: ProbeDown (gitlab ssh ipv6) as Resolved.

This is resolved after switching to monitor IPv4 only. In T367021 I'll discuss more how to properly add IPv6 to the gitlab-ssh service.

Mon, Jun 10, 11:32 AM · collaboration-services
Jelto added a parent task for T367038: ProbeDown (gitlab ssh ipv6): T367021: Create an SSH blackbox test for GitLab.
Mon, Jun 10, 10:35 AM · collaboration-services
Jelto added a subtask for T367021: Create an SSH blackbox test for GitLab: T367038: ProbeDown (gitlab ssh ipv6).
Mon, Jun 10, 10:35 AM · GitLab, collaboration-services
Jelto renamed T367038: ProbeDown (gitlab ssh ipv6) from ProbeDown to ProbeDown (gitlab ssh ipv6).
Mon, Jun 10, 10:35 AM · collaboration-services
Jelto claimed T367021: Create an SSH blackbox test for GitLab.
Mon, Jun 10, 8:34 AM · GitLab, collaboration-services
Jelto moved T367021: Create an SSH blackbox test for GitLab from Incoming to Work in Progress on the collaboration-services board.
Mon, Jun 10, 8:34 AM · GitLab, collaboration-services

Fri, Jun 7

Jelto updated the task description for T366882: Move GitLab behind the CDN.
Fri, Jun 7, 9:00 AM · Patch-For-Review, Release-Engineering-Team, Traffic, collaboration-services, SRE
Jelto created T366882: Move GitLab behind the CDN.
Fri, Jun 7, 8:39 AM · Patch-For-Review, Release-Engineering-Team, Traffic, collaboration-services, SRE

Thu, Jun 6

Jelto lowered the priority of T366786: GitLabRunnerTrustedConfigMissing from High to Medium.

One end-user edited the runner within their project to solve a building/dependency issue:

Thu, Jun 6, 11:52 AM · Patch-For-Review, collaboration-services
Jelto moved T366786: GitLabRunnerTrustedConfigMissing from Incoming to Work in Progress on the collaboration-services board.
Thu, Jun 6, 10:34 AM · Patch-For-Review, collaboration-services
Jelto claimed T366786: GitLabRunnerTrustedConfigMissing.

the alert is right, the protected checkbox was missing. I manually protected the runner again in https://gitlab.wikimedia.org/admin/runners/1484#/.

Thu, Jun 6, 10:34 AM · Patch-For-Review, collaboration-services

Wed, Jun 5

Jelto edited projects for T364839: Investigate / remove custom downstream changes to PhabricatorClientRateLimit.php, added: collaboration-services; removed serviceops.
Wed, Jun 5, 8:48 AM · Phabricator, collaboration-services, Technical-Debt
Jelto added a comment to T364839: Investigate / remove custom downstream changes to PhabricatorClientRateLimit.php.

I left a comment in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/52#note_86778. +1 for removing the custom rate limiting code. I'd try to move any existing extra logic to a single place (requestctl).

Wed, Jun 5, 8:48 AM · Phabricator, collaboration-services, Technical-Debt

Tue, Jun 4

Jelto added a comment to T365675: Upgrade GitLab to major version 17.

We could probably bump https://gitlab.devtools.wmcloud.org/ to v17...

Tue, Jun 4, 9:52 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services

Mon, Jun 3

Jelto closed T365768: Upgrade Gitlab embedded version of Postgres as Resolved.

I executed the cleanup command on all instances, I'll resolve the task.

Mon, Jun 3, 12:52 PM · collaboration-services
Jelto updated the task description for T365768: Upgrade Gitlab embedded version of Postgres.
Mon, Jun 3, 12:51 PM · collaboration-services
Jelto updated subscribers of T365675: Upgrade GitLab to major version 17.

Can't speak for other tools, but Kokkuri uses the new id tokens instead of the old deprecated JWT variables. So at least I can say that Kokkuri is not blocking the upgrade.

For what it's worth, I've done some searching in GitLab and the only repos that I could find that are still using CI_JOB_JWT* tokens are an old fork of Kokkuri and two playgrounds.

A different issue is the deprecation of the /-/jwks endpoint, I know of at least two repos that are still using that, I've created MRs to update them. As in the case of the tokens, I'm not aware of whether we have other repos using the old endpoint:

  1. https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/78
  2. https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/17
Mon, Jun 3, 10:36 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Mon, Jun 3, 10:29 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services

Fri, May 31

Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Fri, May 31, 1:15 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto added a comment to T365768: Upgrade Gitlab embedded version of Postgres.

All instance updated to new postgres version.
I'd move the cleanup (mentioned in the scripts output) to somewhere next week:

Fri, May 31, 11:52 AM · collaboration-services
Jelto updated the task description for T365768: Upgrade Gitlab embedded version of Postgres.
Fri, May 31, 11:51 AM · collaboration-services
Jelto updated the task description for T365768: Upgrade Gitlab embedded version of Postgres.
Fri, May 31, 8:27 AM · collaboration-services
Jelto updated the task description for T365768: Upgrade Gitlab embedded version of Postgres.
Fri, May 31, 8:23 AM · collaboration-services

Thu, May 30

Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Thu, May 30, 1:33 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto closed T366076: SystemdUnitFailed - gitlab1003 as Resolved.

This is fixed now, the exporter is retrying:

Thu, May 30, 8:26 AM · collaboration-services

Wed, May 29

Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Wed, May 29, 3:52 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto added a project to T354887: Upgrade to Gerrit 3.9: collaboration-services.
Wed, May 29, 11:55 AM · collaboration-services, Gerrit (Gerrit 3.9), Release-Engineering-Team
Jelto updated subscribers of T365675: Upgrade GitLab to major version 17.

@jnuche just to be sure, you replaced the old versions of JSON web tokens (CI_JOB_JWT*) in T337474 and we should not be affected by the deprecation anymore?

Wed, May 29, 10:37 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Wed, May 29, 10:34 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Wed, May 29, 10:02 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto closed T366087: SystemdUnitFailed - gitlab1003 - gitlab-exporter, a subtask of T354656: Create a custom GitLab Prometheus exporter, as Resolved.
Wed, May 29, 7:47 AM · Wikimedia-Hackathon-2024, GitLab (Infrastructure), collaboration-services
Jelto closed T366087: SystemdUnitFailed - gitlab1003 - gitlab-exporter as Resolved.

This was caused by the daily backup-restore. In v1.0.10 of the exporter I added better error handling : https://gitlab.wikimedia.org/repos/sre/gitlab-exporter/-/tags/v1.0.10

Wed, May 29, 7:47 AM · collaboration-services
Jelto merged T366131: SystemdUnitFailed - gitlab1003, gitlab1004 - gitlab-exporter into T366087: SystemdUnitFailed - gitlab1003 - gitlab-exporter.
Wed, May 29, 7:45 AM · collaboration-services
Jelto merged task T366131: SystemdUnitFailed - gitlab1003, gitlab1004 - gitlab-exporter into T366087: SystemdUnitFailed - gitlab1003 - gitlab-exporter.
Wed, May 29, 7:45 AM · collaboration-services

Tue, May 28

thcipriani awarded T354656: Create a custom GitLab Prometheus exporter a Like token.
Tue, May 28, 9:32 PM · Wikimedia-Hackathon-2024, GitLab (Infrastructure), collaboration-services
Jelto claimed T366076: SystemdUnitFailed - gitlab1003.

I can upload a fix for that in the exporter tomorrow

Tue, May 28, 3:20 PM · collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Tue, May 28, 1:55 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Tue, May 28, 1:54 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Tue, May 28, 1:39 PM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Tue, May 28, 11:47 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto updated the task description for T365675: Upgrade GitLab to major version 17.
Tue, May 28, 11:45 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto triaged T365675: Upgrade GitLab to major version 17 as High priority.
Tue, May 28, 11:39 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto closed T354656: Create a custom GitLab Prometheus exporter as Resolved.

The exporter is running now on all GitLab instances and an alert for missing config for the Trusted Runners is active. I added some visualizations for the new metrics:

Tue, May 28, 10:15 AM · Wikimedia-Hackathon-2024, GitLab (Infrastructure), collaboration-services
Jelto moved T365675: Upgrade GitLab to major version 17 from Incoming to Work in Progress on the collaboration-services board.
Tue, May 28, 10:13 AM · Release-Engineering-Team, GitLab (Infrastructure), collaboration-services
Jelto closed Restricted Task, a subtask of T84: Make sure anti-vandalism features are up to snuff, as Resolved.
Tue, May 28, 8:14 AM · Phabricator, Wikimedia Phabricator RfC

Fri, May 24

Jelto added a comment to T354656: Create a custom GitLab Prometheus exporter.

The alert with a quick test on one of the replicas works, see T365802. I'll do some more documentation and then close the task.

Fri, May 24, 12:15 PM · Wikimedia-Hackathon-2024, GitLab (Infrastructure), collaboration-services
Jelto added a subtask for T354656: Create a custom GitLab Prometheus exporter: T365802: GitLabRunnerTrustedConfigMissing.
Fri, May 24, 12:11 PM · Wikimedia-Hackathon-2024, GitLab (Infrastructure), collaboration-services
Jelto added a parent task for T365802: GitLabRunnerTrustedConfigMissing: T354656: Create a custom GitLab Prometheus exporter.
Fri, May 24, 12:11 PM · collaboration-services
Jelto closed T365802: GitLabRunnerTrustedConfigMissing as Resolved.

this was a test for T365802 on one of the replicas

Fri, May 24, 12:11 PM · collaboration-services
Jelto added a comment to T354656: Create a custom GitLab Prometheus exporter.

Unfortunately the python clients for the gitlab api and prometheus are outdated in the Bullseye Debian package. So I had to implement some workarounds in the exporter for missing features/api endpoints.

Fri, May 24, 11:54 AM · Wikimedia-Hackathon-2024, GitLab (Infrastructure), collaboration-services
Jelto added a parent task for T365800: ProbeDown (gitlab2002): Unknown Object (Task).
Fri, May 24, 11:08 AM · collaboration-services
Jelto closed T365800: ProbeDown (gitlab2002) as Resolved.

happened during upgrade in T365587

Fri, May 24, 11:08 AM · collaboration-services
Jelto renamed T365800: ProbeDown (gitlab2002) from ProbeDown to ProbeDown (gitlab2002).
Fri, May 24, 11:07 AM · collaboration-services
Jelto added a subtask for T283615: Make mailman3 work in the standby host (lists2001.wikimedia.org): T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.
Fri, May 24, 7:32 AM · SRE, Datacenter-Switchover, Wikimedia-Mailing-lists
Jelto removed a subtask for T331706: Migrate Mailman/lists to Bullseye/Bookworm: T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.
Fri, May 24, 7:32 AM · Patch-For-Review, collaboration-services, Wikimedia-Mailing-lists, SRE
Jelto edited parent tasks for T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001, added: T283615: Make mailman3 work in the standby host (lists2001.wikimedia.org); removed: T331706: Migrate Mailman/lists to Bullseye/Bookworm.
Fri, May 24, 7:32 AM · collaboration-services
Jelto added a subtask for T331706: Migrate Mailman/lists to Bullseye/Bookworm: T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.
Fri, May 24, 7:31 AM · Patch-For-Review, collaboration-services, Wikimedia-Mailing-lists, SRE
Jelto added a parent task for T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001: T331706: Migrate Mailman/lists to Bullseye/Bookworm.
Fri, May 24, 7:31 AM · collaboration-services
Jelto merged T365722: SystemdUnitFailed - backup-restore.service - gitlab1003 into T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.
Fri, May 24, 7:31 AM · collaboration-services
Jelto merged task T365722: SystemdUnitFailed - backup-restore.service - gitlab1003 into T365781: SystemdUnitFailed - gitlab1003, gitlab1004, lists2001.
Fri, May 24, 7:31 AM · collaboration-services