Page MenuHomePhabricator
Create Task
Maniphest T409835

Apt-staging: add alerting
Closed, ResolvedPublic
Actions

Description

Follow up from T409253: Continuous breakages of apt-staging

Right now we have no alerting on failed imports, relying on the users to tell us when there is a problem. We should add alerting to be aware of issues with the import pipeline.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
apt-staging: alert only on codfwoperations/alertsmaster+1 -1
apt-staging: wrong error code in gitlab_package_puller_run_successoperations/alertsmaster+2 -2
apt-staging: logging and metricsoperations/puppetproduction+261 -47
apt: add an alert on reprepro errorsoperations/alertsmaster+78 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

LSobanski created this task.Nov 11 2025, 2:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 11 2025, 2:19 PM
MoritzMuehlenhoff subscribed.Nov 11 2025, 2:20 PM
ABran-WMF subscribed.Nov 14 2025, 8:58 AM
gerritbot added a comment.Nov 14 2025, 2:09 PM

Change #1205162 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] apt-staging: logging and metrics

https://gerrit.wikimedia.org/r/1205162

gerritbot added a project: Patch-For-Review.Nov 14 2025, 2:09 PM
LSobanski assigned this task to ABran-WMF.Nov 17 2025, 12:22 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.
ABran-WMF added a comment.Nov 18 2025, 7:50 AM

basic metrics are visible in T409833#11382292:

# HELP gitlab_package_puller_jobs_considered Number of CI jobs inspected in the last run
# TYPE gitlab_package_puller_jobs_considered gauge
gitlab_package_puller_jobs_considered 2566
# HELP gitlab_package_puller_jobs_downloaded Number of CI jobs whose artifacts were downloaded in the last run
# TYPE gitlab_package_puller_jobs_downloaded gauge
gitlab_package_puller_jobs_downloaded 8
# HELP gitlab_package_puller_run_success Whether the last run completed without unhandled exceptions (1=success, 0=failure)
# TYPE gitlab_package_puller_run_success gauge
gitlab_package_puller_run_success 1
# HELP gitlab_package_puller_last_run_timestamp_seconds Unix timestamp of the end of the last run
# TYPE gitlab_package_puller_last_run_timestamp_seconds gauge
gitlab_package_puller_last_run_timestamp_seconds 1763451717

With T409832: Apt-staging: add error handling to gitlab_package_puller I'll add some more

ABran-WMF added a subtask: T409832: Apt-staging: add error handling to gitlab_package_puller.Nov 18 2025, 7:50 AM
ABran-WMF changed the status of subtask T409832: Apt-staging: add error handling to gitlab_package_puller from Open to In Progress.
gerritbot added a comment.Nov 20 2025, 8:57 AM

Change #1207791 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/alerts@master] apt: add an alert on reprepro errors

https://gerrit.wikimedia.org/r/1207791

ABran-WMF added a comment.Nov 20 2025, 10:36 AM

I've added alerts on reprepro more general script failures in operations/alerts/+/1207791

It'll also be possible to add another notification medium if needed

gerritbot added a comment.Nov 24 2025, 7:01 AM

Change #1207791 merged by jenkins-bot:

[operations/alerts@master] apt: add an alert on reprepro errors

https://gerrit.wikimedia.org/r/1207791

gerritbot added a comment.Nov 25 2025, 7:16 AM

Change #1205162 merged by Arnaudb:

[operations/puppet@production] apt-staging: logging and metrics

https://gerrit.wikimedia.org/r/1205162

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2025, 7:31 AM
ABran-WMF closed this task as Resolved.Nov 25 2025, 8:01 AM

as mentioned in T409833: Apt-staging: fix logging

In T409833#11404142, @ABran-WMF wrote:

We now have a bit more observability via metrics [...]:

root@apt-staging2001:node.d $ cat gitlab_package_puller.prom 
[...]
gitlab_package_puller_run_success 1
[...]
gitlab_package_puller_reprepro_notify_failed 0

[...]

A simple set of alerts is now in production

gerritbot added a comment.Nov 25 2025, 8:29 AM

Change #1211001 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/alerts@master] apt-staging: wrong error code in gitlab_package_puller_run_success

https://gerrit.wikimedia.org/r/1211001

gerritbot added a project: Patch-For-Review.Nov 25 2025, 8:29 AM
gerritbot added a comment.Nov 25 2025, 8:35 AM

Change #1211001 merged by jenkins-bot:

[operations/alerts@master] apt-staging: wrong error code in gitlab_package_puller_run_success

https://gerrit.wikimedia.org/r/1211001

ABran-WMF mentioned this in T410984: GitlabPackagePullerFailedOnRun.Nov 25 2025, 8:41 AM
ABran-WMF added a subtask: T410984: GitlabPackagePullerFailedOnRun.
Maintenance_bot removed a project: Patch-For-Review.Nov 25 2025, 9:31 AM
Jelto reopened this task as Open.Dec 5 2025, 2:25 PM
Jelto subscribed.

There is an open alert since 11 days Linting problems found for GitlabPackagePullerFailedOnRun:

description: Pint reporter promql/series found problem(s) in /srv/alerts/ops/team-sre_apt.yaml: prometheus "ops" at http://127.0.0.1:9900/ops didn't have any series for "gitlab_package_puller_reprepro_notify_failed" metric in the last 1w
summary: Linting problems found for GitlabPackagePullerFailedOnReprepro

https://alerts.wikimedia.org/?q=%40state%3Dactive&q=alertname%3DAlertLintProblem&q=filename%3D%2Fsrv%2Falerts%2Fops%2Fteam-sre_apt.yaml

I'm not fully sure why this is happening, thanos shows a series with that name. The summary mentions GitlabPackagePullerFailedOnRun which is a test added in https://gerrit.wikimedia.org/r/c/operations/alerts/+/1211001.

So I'll reopen the task until that alert test is fixed.

ABran-WMF added a project: Observability-Alerting.Dec 8 2025, 8:55 AM
In T409835#11436479, @Jelto wrote:

There is an open alert since 11 days Linting problems found for GitlabPackagePullerFailedOnRun:

I'm not fully sure why this is happening, thanos shows a series with that name.

I can confirm the metric is also present on the host:

root@apt-staging2001:node.d $ rg gitlab_package_puller_reprepro_notify_failed gitlab_package_puller.prom
25:# HELP gitlab_package_puller_reprepro_notify_failed Number of times a reprepro failure notification was (or would have been) triggered in the last run
26:# TYPE gitlab_package_puller_reprepro_notify_failed gauge
27:gitlab_package_puller_reprepro_notify_failed 0
root@apt-staging2001:node.d $ pwd
/var/lib/prometheus/node.d

I'll ask Observability for help

gerritbot added a comment.Dec 8 2025, 9:04 AM

Change #1216549 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/alerts@master] apt-staging: alert only on codfw

https://gerrit.wikimedia.org/r/1216549

gerritbot added a project: Patch-For-Review.Dec 8 2025, 9:04 AM
gerritbot added a comment.Dec 8 2025, 9:13 AM

Change #1216549 merged by jenkins-bot:

[operations/alerts@master] apt-staging: alert only on codfw

https://gerrit.wikimedia.org/r/1216549

ABran-WMF closed this task as Resolved.Dec 8 2025, 9:27 AM

I mistakenly deployed these alerts on eqiad as well

Maintenance_bot removed a project: Patch-For-Review.Dec 8 2025, 9:30 AM
ABran-WMF closed subtask T409832: Apt-staging: add error handling to gitlab_package_puller as Resolved.Dec 9 2025, 7:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits