Page MenuHomePhabricator

SystemdUnitFailed - gitlab-settings-configure-projects.service on gitlab1004:9100
Closed, ResolvedPublic

Description

Common information

  • alertname: SystemdUnitFailed
  • instance: gitlab1004:9100
  • prometheus: ops
  • severity: critical
  • site: eqiad
  • source: prometheus
  • team: collaboration-services

Firing alerts



Event Timeline

ABran-WMF renamed this task from SystemdUnitFailed to SystemdUnitFailed - gitlab-settings-configure-projects.service on gitlab1004:9100.Jun 8 2026, 6:36 AM
ABran-WMF subscribed.
Jun 08 06:35:16 gitlab1004 systemd[1]: Starting gitlab-settings-configure-projects.service - Configure all projects on instance...
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,459 dw31415/simple-rfc-stats - setting squash commit template
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,459 dw31415/simple-rfc-stats - update automatic pipeline cleanup time
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,459 dw31415/simple-rfc-stats - saving new settings
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,608 Unable to save project. 500: 500 Internal Server Error
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,910 sd/produnto-test-deletion_scheduled-4615 - setting squash commit template
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,910 sd/produnto-test-deletion_scheduled-4615 - update automatic pipeline cleanup time
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,910 sd/produnto-test-deletion_scheduled-4615 - saving new settings
Jun 08 06:35:19 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:19,979 Unable to save project. 403: 403 Forbidden
Jun 08 06:35:20 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:20,290 chrisalbon/test-deletion_scheduled-4608 - setting squash commit template
Jun 08 06:35:20 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:20,291 chrisalbon/test-deletion_scheduled-4608 - update automatic pipeline cleanup time
Jun 08 06:35:20 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:20,291 chrisalbon/test-deletion_scheduled-4608 - saving new settings
Jun 08 06:35:20 gitlab1004 configure-projects[299029]: 2026-06-08 06:35:20,349 Unable to save project. 403: 403 Forbidden
Jun 08 06:35:45 gitlab1004 configure-projects[299029]: Traceback (most recent call last):
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/srv/gitlab-settings/configure-projects/configure-projects", line 184, in <module>
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     main()
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/srv/gitlab-settings/configure-projects/configure-projects", line 85, in main
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     for project in get_projects(config):
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/usr/lib/python3/dist-packages/gitlab/base.py", line 295, in __next__
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     return self.next()
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:            ^^^^^^^^^^^
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/usr/lib/python3/dist-packages/gitlab/base.py", line 298, in next
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     data = self._list.next()
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:            ^^^^^^^^^^^^^^^^^
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/usr/lib/python3/dist-packages/gitlab/client.py", line 1244, in next
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     self._query(self._next_url, **self._kwargs)
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/usr/lib/python3/dist-packages/gitlab/client.py", line 1156, in _query
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     result = self._gl.http_request("get", url, query_data=query_data, **kwargs)
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:   File "/usr/lib/python3/dist-packages/gitlab/client.py", line 800, in http_request
Jun 08 06:35:45 gitlab1004 configure-projects[299029]:     raise gitlab.exceptions.GitlabHttpError(
Jun 08 06:35:45 gitlab1004 configure-projects[299029]: gitlab.exceptions.GitlabHttpError: 500: 500 Internal Server Error
Jun 08 06:35:45 gitlab1004 systemd[1]: gitlab-settings-configure-projects.service: Main process exited, code=exited, status=1/FAILURE
$ sudo systemctl restart --no-block gitlab-settings-configure-projects.service  && journalctl -fln0 -u gitlab-settings-configure-projects.service
Jun 08 09:29:20 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:20,709 sd/produnto-test-deletion_scheduled-4615 - setting squash commit template
Jun 08 09:29:20 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:20,709 sd/produnto-test-deletion_scheduled-4615 - update automatic pipeline cleanup time
Jun 08 09:29:20 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:20,709 sd/produnto-test-deletion_scheduled-4615 - saving new settings
Jun 08 09:29:20 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:20,772 Unable to save project. 403: 403 Forbidden
Jun 08 09:29:21 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:21,080 chrisalbon/test-deletion_scheduled-4608 - setting squash commit template
Jun 08 09:29:21 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:21,080 chrisalbon/test-deletion_scheduled-4608 - update automatic pipeline cleanup time
Jun 08 09:29:21 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:21,080 chrisalbon/test-deletion_scheduled-4608 - saving new settings
Jun 08 09:29:21 gitlab1004 configure-projects[18295]: 2026-06-08 09:29:21,148 Unable to save project. 403: 403 Forbidden
Jun 08 09:32:46 gitlab1004 systemd[1]: gitlab-settings-configure-projects.service: Deactivated successfully.
Jun 08 09:32:46 gitlab1004 systemd[1]: Finished gitlab-settings-configure-projects.service - Configure all projects on instance.
Jun 08 09:32:46 gitlab1004 systemd[1]: gitlab-settings-configure-projects.service: Consumed 2.688s CPU time.