User Details
User Details
- User Since
- Aug 29 2023, 8:30 AM (118 w, 4 d)
- Availability
- Available
- IRC Nick
- arnaudb
- LDAP User
- Arnaudb
- MediaWiki User
- ABran-WMF [ Global Accounts ]
Wed, Dec 3
Wed, Dec 3
Fri, Nov 28
Fri, Nov 28
ABran-WMF updated subscribers of T411240: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100.
@Jelto do you think it could be worth implementing a check of this service at the end of the upgrade job on a primary gitlab instance?
ABran-WMF added a parent task for T411240: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100: Unknown Object (Task).
ABran-WMF renamed T411240: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100 from SystemdUnitFailed to SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100.
Thu, Nov 27
Thu, Nov 27
ABran-WMF closed T411197: SystemdUnitFailed - backup-restore.service on gitlab2002:9100 as Resolved.
ABran-WMF added a parent task for T411197: SystemdUnitFailed - backup-restore.service on gitlab2002:9100: Unknown Object (Task).
ABran-WMF renamed T411197: SystemdUnitFailed - backup-restore.service on gitlab2002:9100 from SystemdUnitFailed to SystemdUnitFailed - backup-restore.service on gitlab2002:9100.
Tue, Nov 25
Tue, Nov 25
ABran-WMF added a subtask for T409835: Apt-staging: add alerting: T410984: GitlabPackagePullerFailedOnRun.
ABran-WMF added a parent task for T410984: GitlabPackagePullerFailedOnRun: T409835: Apt-staging: add alerting.
error from T409835: Apt-staging: add alerting and fixed via https://gerrit.wikimedia.org/r/c/operations/alerts/+/1211001
ABran-WMF updated subscribers of T409832: Apt-staging: add error handling to gitlab_package_puller.
@MoritzMuehlenhoff as @Dzahn mentioned, you might be interested to review what's been done on the gitlab-package-puller script. I'd be happy to have a chat about it if needed, let me know if you see any missing things or any room for improvement
as mentioned in T409833: Apt-staging: fix logging
We now have a bit more observability via metrics and logging:
Mon, Nov 24
Mon, Nov 24
ABran-WMF added a comment to T370677: migrate all sre-collab services to nftables.
docker has released a test version which now supports nftables
Thu, Nov 20
Thu, Nov 20
ABran-WMF added a comment to T409835: Apt-staging: add alerting.
I've added alerts on reprepro more general script failures in operations/alerts/+/1207791
Tue, Nov 18
Tue, Nov 18
ABran-WMF changed the status of T409832: Apt-staging: add error handling to gitlab_package_puller, a subtask of T409833: Apt-staging: fix logging, from Open to In Progress.
ABran-WMF changed the status of T409832: Apt-staging: add error handling to gitlab_package_puller, a subtask of T409835: Apt-staging: add alerting, from Open to In Progress.
ABran-WMF changed the status of T409832: Apt-staging: add error handling to gitlab_package_puller from Open to In Progress.
ABran-WMF added a parent task for T409832: Apt-staging: add error handling to gitlab_package_puller: T409835: Apt-staging: add alerting.
ABran-WMF added a comment to T409835: Apt-staging: add alerting.
basic metrics are visible in T409833#11382292:
The updated version of the log output is a bit more verbose at info level:
{P85360}
Mon, Nov 17
Mon, Nov 17
ABran-WMF added a comment to T409833: Apt-staging: fix logging.
T409832: Apt-staging: add error handling to gitlab_package_puller could be bundled with this task
ABran-WMF added a parent task for T409832: Apt-staging: add error handling to gitlab_package_puller: T409833: Apt-staging: fix logging.
Fri, Nov 14
Fri, Nov 14
ABran-WMF added a parent task for T410044: SystemdUnitFailed - gitlab2002 - backup-restore: Unknown Object (Task).
Oct 17 2025
Oct 17 2025
ABran-WMF closed T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Resolved.
I think that can be closed now, feel free to reopen if needed!
ABran-WMF added a comment to T387833: Gerrit failover process.
Oct 15 2025
Oct 15 2025
ABran-WMF moved T407312: ProbeDown - gerrit1003:443 from Incoming to Work in Progress on the collaboration-services board.
related to {T365259}
Oct 14 2025
Oct 14 2025
same pattern as the one I was trying to fix in https://phabricator.wikimedia.org/T406403#11253645 → https://grafana.wikimedia.org/goto/ENnpKa6NR?orgId=1
This will be fixed by {T365259}
Oct 13 2025
Oct 13 2025
ABran-WMF updated the task description for T387833: Gerrit failover process.
ABran-WMF added a comment to T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space.
Given T387833#11267438, that should not be an issue anymore, I'll send a patch to re-enable backups
ABran-WMF added a comment to T387833: Gerrit failover process.
things are looking better now:
arnaudb@gerrit2003:git $ fd | wc -l 236218 arnaudb@gerrit2003:git $ pwd /srv/gerrit/git arnaudb@gerrit2003:git $ ls -l /srv/backup/ total 0
vs the previous situation:
Oct 9 2025
Oct 9 2025
ABran-WMF added a parent task for T406856: Reduce size of analytics/superset/deploy.git Gerrit repo: T387833: Gerrit failover process.
ABran-WMF updated the task description for T387833: Gerrit failover process.
Oct 8 2025
Oct 8 2025
ABran-WMF renamed T404965: gerrit: fix allocstalls from Gerrit: fix allocstalls to gerrit: fix allocstalls.
Oct 7 2025
Oct 7 2025
ABran-WMF updated the task description for T387833: Gerrit failover process.
ABran-WMF added a comment to T387833: Gerrit failover process.
DRY-RUN: Executing commands ['/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit2/review_site rsync://gerrit2003.wikimedia.org/gerrit-var-lib/ --no-o --no-g --chown=gerrit:gerrit '] on 1 hosts: gerrit1003.wikimedia.org DRY-RUN: Releasing lock for key sre.gerrit.failover with ID abe63737-d8ed-498c-af9d-f71d5fe4d64c
ABran-WMF renamed T406546: SystemdUnitFailed - exim4 and wmf_auto_restart_exim4 on lists1004 from SystemdUnitFailed to SystemdUnitFailed - exim4 and wmf_auto_restart_exim4 on lists1004.
ABran-WMF closed T406546: SystemdUnitFailed - exim4 and wmf_auto_restart_exim4 on lists1004, a subtask of T378021: Replace Exim on lists.wikimedia.org with Postfix, as Resolved.
ABran-WMF closed T406546: SystemdUnitFailed - exim4 and wmf_auto_restart_exim4 on lists1004 as Resolved.
forced a stop/start of exim4 and wmf_auto_restart_exim4.service
Oct 6 2025
Oct 6 2025
ABran-WMF added a comment to T387833: Gerrit failover process.
thanks for that @Jelto I'll try and reproduce the error in a controlled environment
ABran-WMF moved T402260: Replace Spamassassin with Rspam for VRTS on Postfix from Work in Progress to Backlog on the collaboration-services board.
Oct 3 2025
Oct 3 2025
ABran-WMF created T406333: gerrit: config tweaks.
Oct 2 2025
Oct 2 2025
ABran-WMF updated the task description for T387833: Gerrit failover process.
ABran-WMF changed the status of T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid, a subtask of T372804: setup gerrit2003 with gerrit service (gerrit on bookworm), from Stalled to In Progress.
ABran-WMF changed the status of T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid, a subtask of T260666: Create a cookbook to automate gerrit's switchover, from Stalled to In Progress.
ABran-WMF changed the status of T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid, a subtask of T387833: Gerrit failover process, from Stalled to In Progress.
ABran-WMF changed the status of T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid from Stalled to In Progress.
good catch, indeed!
ABran-WMF updated the task description for T387833: Gerrit failover process.
ABran-WMF updated the task description for T387833: Gerrit failover process.
Oct 1 2025
Oct 1 2025
ABran-WMF updated the task description for T406017: gerrit: mod_qos allowlist and monitoring improvements.
ABran-WMF moved T406017: gerrit: mod_qos allowlist and monitoring improvements from Incoming to Work in Progress on the collaboration-services board.
mtail was not the source of the observability issue. I've fixed the promql query that renders the QoS event rates using this uptick as reference from the logs.
ABran-WMF updated the task description for T406017: gerrit: mod_qos allowlist and monitoring improvements.
Sep 30 2025
Sep 30 2025
ABran-WMF closed T406005: train-presync failed due to git clone failing with gnutls_handshake() failure, a subtask of T405677: 1.45.0-wmf.21 deployment blockers, as Resolved.
ABran-WMF closed T406005: train-presync failed due to git clone failing with gnutls_handshake() failure as Resolved.
This was a side effect of {T402611}, it's been worked around for now
ABran-WMF added a comment to T406005: train-presync failed due to git clone failing with gnutls_handshake() failure.
change applied on gerrit1003, it should be back to normal now
ABran-WMF added a comment to T406005: train-presync failed due to git clone failing with gnutls_handshake() failure.
I think I found the issue:
Sep 29 2025
Sep 29 2025
ABran-WMF updated the task description for T387833: Gerrit failover process.
ABran-WMF added a comment to T387833: Gerrit failover process.
the dry run switchover output looks alright {P83480}
CI went through, thanks for the fix!
Sep 26 2025
Sep 26 2025
ABran-WMF updated subscribers of T405706: CI error on operations/cookbooks.
ABran-WMF updated the task description for T387833: Gerrit failover process.
Sep 24 2025
Sep 24 2025
ABran-WMF added a comment to T405455: SystemdUnitFailed - wmf_auto_restart_exim4.service on lists1004.
root@lists1004:~ $ sudo systemctl stop exim4 root@lists1004:~ $ ps auxf|rg -i exim mtail 836 0.4 0.0 3308356 27304 ? Ssl May26 719:22 /usr/bin/mtail --progs /etc/mtail --logtostderr --address :: --port 3903 --logs /var/log/exim4/mainlog,/var/log/mailman/smtp,/var/log/mailman/subscribe -disable_fsnotify root 1201963 0.0 0.0 9124 5960 pts/0 S+ 10:20 0:00 \_ rg -i exim root 1193401 0.0 0.0 47848 27964 ? S 10:14 0:00 /usr/sbin/exim4 -q root 1193491 0.0 0.0 47848 22400 ? S 10:14 0:00 \_ /usr/sbin/exim4 -q Debian-+ 1193492 0.0 0.0 48004 24288 ? S 10:14 0:00 \_ /usr/sbin/exim4 -q root 1200450 0.8 0.0 47840 28080 ? S 10:19 0:00 /usr/sbin/exim4 -q root 1201885 0.0 0.0 47840 22524 ? S 10:20 0:00 \_ /usr/sbin/exim4 -q Debian-+ 1201886 0.0 0.0 47844 23988 ? S 10:20 0:00 \_ /usr/sbin/exim4 -q root 1201639 1.2 0.0 47844 28008 ? S 10:20 0:00 /usr/sbin/exim4 -q Debian-+ 1201964 0.0 0.0 0 0 ? R 10:20 0:00 \_ [exim4]
ABran-WMF renamed T405455: SystemdUnitFailed - wmf_auto_restart_exim4.service on lists1004 from SystemdUnitFailed to SystemdUnitFailed - wmf_auto_restart_exim4.service on lists1004.