Page MenuHomePhabricator

SystemdUnitFailed - gitlab1004 - sync-gitlab-group-with-ldap
Closed, ResolvedPublic

Description

Common information

  • alertname: SystemdUnitFailed
  • instance: gitlab1004:9100
  • name: sync-gitlab-group-with-ldap.service
  • prometheus: ops
  • severity: critical
  • site: eqiad
  • source: prometheus
  • team: collaboration-services

Firing alerts


Event Timeline

LSobanski renamed this task from SystemdUnitFailed to SystemdUnitFailed - gitlab1004 - sync-gitlab-group-with-ldap.Wed, May 20, 6:30 AM
ABran-WMF claimed this task.
ABran-WMF subscribed.

last run was:

May 20 08:15:03 gitlab1004 systemd[1]: Starting sync-gitlab-group-with-ldap.service - Sync various GitLab groups with their LDAP groups...
May 20 08:15:04 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:04,096 Collecting membership list of LDAP group wmf
May 20 08:15:04 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:04,123 Collecting membership list of LDAP group ops
May 20 08:15:04 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:04,126 Collecting member list of Gitlab group repos/mediawiki
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,398 ldap user mfischerwmf needs to be added to repos/mediawiki.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,442 user mfischerwmf will be created in Gitlab.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,443 ldap user yliou needs to be added to repos/mediawiki.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,489 user yliou will be created in Gitlab.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,490 ldap user bmartinez needs to be added to repos/mediawiki.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,546 user bmartinez will be created in Gitlab.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,546 ldap user mbinder needs to be added to repos/mediawiki.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,594 user mbinder will be created in Gitlab.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,595 There are 4 GitLab users to create.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,595 There are 4 members to add to repos/mediawiki.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,596 There are 0 members to remove from repos/mediawiki.
May 20 08:15:07 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:07,673 Creating gitlab user mfischerwmf
May 20 08:15:08 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:08,286 Failed to create user mfischerwmf with email address mfischer@wikimedia.org: Email has already been taken
May 20 08:15:08 gitlab1004 sync-gitlab-group-with-ldap[2503169]: Skipping.
May 20 08:15:08 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:08,388 The conflicting address is held by blueray: https://gitlab.wikimedia.org/blueray
May 20 08:15:08 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:08,450 Creating gitlab user yliou
May 20 08:15:09 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:09,074 Failed to create user yliou with email address yliou@wikimedia.org: Email has already been taken
May 20 08:15:09 gitlab1004 sync-gitlab-group-with-ldap[2503169]: Skipping.
May 20 08:15:09 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:09,297 The conflicting address is held by yumingliou: https://gitlab.wikimedia.org/yumingliou
May 20 08:15:09 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:09,370 Creating gitlab user bmartinez
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,008 Failed to create user bmartinez with email address bmartinezcalvo@wikimedia.org: Email has already been taken
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: Skipping.
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,111 The conflicting address is held by bmartinezcalvo: https://gitlab.wikimedia.org/bmartinezcalvo
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,187 Creating gitlab user mbinder
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,795 Failed to create user mbinder with email address mbinder@wikimedia.org: Email has already been taken
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: Skipping.
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,932 The conflicting address is held by maxbinderwmf: https://gitlab.wikimedia.org/maxbinderwmf
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,932 Failed to create 4 users.
May 20 08:15:10 gitlab1004 sync-gitlab-group-with-ldap[2503169]: 2026-05-20 08:15:10,932 Sync completed.
May 20 08:15:11 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:11,541 Collecting membership list of LDAP group ops
May 20 08:15:11 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:11,581 Collecting member list of Gitlab group repos/sre
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:12,189 There are 0 GitLab users to create.
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:12,189 There are 0 members to add to repos/sre.
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:12,189 There are 0 members to remove from repos/sre.
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:12,189 No work to do.
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2505845]: 2026-05-20 08:15:12,189 Sync completed.
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:12,723 Collecting membership list of LDAP group wmde
May 20 08:15:12 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:12,755 Collecting member list of Gitlab group people/wmde
May 20 08:15:13 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:13,253 There are 0 GitLab users to create.
May 20 08:15:13 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:13,253 There are 0 members to add to people/wmde.
May 20 08:15:13 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:13,253 There are 0 members to remove from people/wmde.
May 20 08:15:13 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:13,253 No work to do.
May 20 08:15:13 gitlab1004 sync-gitlab-group-with-ldap[2506384]: 2026-05-20 08:15:13,253 Sync completed.
May 20 08:15:13 gitlab1004 systemd[1]: sync-gitlab-group-with-ldap.service: Deactivated successfully.

where the error triggering the systemd alert was:

May 20 06:30:10 gitlab1004 sync-gitlab-group-with-ldap[212938]: 2026-05-20 06:30:10,333 The conflicting address is held by blueray: https://gitlab.wikimedia.org/blueray
May 20 06:30:10 gitlab1004 sync-gitlab-group-with-ldap[212938]: 2026-05-20 06:30:10,333 Failed to create 4 users.
May 20 06:30:10 gitlab1004 sync-gitlab-group-with-ldap[212938]: 2026-05-20 06:30:10,333 Sync completed.
May 20 06:30:10 gitlab1004 sync-gitlab-group-with-ldap[215605]: 2026-05-20 06:30:10,880 Collecting membership list of LDAP group ops
May 20 06:30:10 gitlab1004 sync-gitlab-group-with-ldap[215605]: 2026-05-20 06:30:10,913 Collecting member list of Gitlab group repos/sre
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]: Traceback (most recent call last):
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:   File "/srv/gitlab-settings/group-management/sync-gitlab-group-with-ldap", line 84, in <module>
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:     main()
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:   File "/srv/gitlab-settings/group-management/sync-gitlab-group-with-ldap", line 43, in main
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:     gitlab_group_users = helper.get_group_members(args.group_path, all=True)
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:   File "/srv/gitlab-settings/group-management/helpers.py", line 170, in get_group_members
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:     return self._collect_users(url)
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:            ^^^^^^^^^^^^^^^^^^^^^^^^
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:   File "/srv/gitlab-settings/group-management/helpers.py", line 93, in _collect_users
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:     for user in self._get_paginated(path):
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:   File "/srv/gitlab-settings/group-management/helpers.py", line 72, in _get_paginated
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:     resp.raise_for_status()
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 1021, in raise_for_status
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]:     raise HTTPError(http_error_msg, response=self)
May 20 06:30:11 gitlab1004 sync-gitlab-group-with-ldap[215605]: requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://gitlab.wikimedia.org/api/v4/groups/repos%2Fsre/members/all?id=repos%2Fsre&page=2&per_page=20

This is tied to T423984: Add retries, error handling and metrics to sync-gitlab-group-with-ldap so I'm marking this as Resolved