ABran-WMF (arnaudb)
SRE

Projects (8)
View All

Calendar

User Details

User Since: Aug 29 2023, 8:30 AM (163 w, 3 d)
Availability: Available
IRC Nick: arnaudb
LDAP User: Arnaudb
MediaWiki User: ABran-WMF [ Global Accounts ]

Recent Activity
View All

Today

ABran-WMF renamed T441394: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100 from SystemdUnitFailed to SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100.

Fri, Oct 16, 9:11 AM · Collaboration-Services

ABran-WMF closed T441394: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100 as Resolved.

arnaudb@gitlab1004:~ $ sudo systemctl restart --no-block sync-gitlab-group-with-ldap.service && journalctl -fln0 -u sync-gitlab-group-with-ldap.service
Oct 16 09:10:27 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:27,707 Collecting membership list of LDAP group wmf
Oct 16 09:10:27 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:27,748 Collecting membership list of LDAP group ops
Oct 16 09:10:27 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:27,751 Collecting member list of Gitlab group repos/mediawiki
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,549 ldap user mfischerwmf needs to be added to repos/mediawiki.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,613 user mfischerwmf will be created in Gitlab.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,613 ldap user bmartinez needs to be added to repos/mediawiki.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,679 user bmartinez will be created in Gitlab.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,679 ldap user mbinder needs to be added to repos/mediawiki.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,744 user mbinder will be created in Gitlab.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,744 ldap user yliou needs to be added to repos/mediawiki.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,839 user yliou will be created in Gitlab.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,840 There are 4 GitLab users to create.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,840 There are 4 members to add to repos/mediawiki.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,840 There are 0 members to remove from repos/mediawiki.
Oct 16 09:10:47 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:47,923 Creating gitlab user mfischerwmf
Oct 16 09:10:48 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:48,563 Failed to create user mfischerwmf with email address mfischer@wikimedia.org: Email has already been taken
Oct 16 09:10:48 gitlab1004 sync-gitlab-group-with-ldap[2655675]: Skipping.
Oct 16 09:10:48 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:48,681 The conflicting address is held by blueray: https://gitlab.wikimedia.org/blueray
Oct 16 09:10:48 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:48,799 Creating gitlab user bmartinez
Oct 16 09:10:49 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:49,434 Failed to create user bmartinez with email address bmartinezcalvo@wikimedia.org: Email has already been taken
Oct 16 09:10:49 gitlab1004 sync-gitlab-group-with-ldap[2655675]: Skipping.
Oct 16 09:10:49 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:49,530 The conflicting address is held by bmartinezcalvo: https://gitlab.wikimedia.org/bmartinezcalvo
Oct 16 09:10:49 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:49,593 Creating gitlab user mbinder
Oct 16 09:10:50 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:50,225 Failed to create user mbinder with email address mbinder@wikimedia.org: Email has already been taken
Oct 16 09:10:50 gitlab1004 sync-gitlab-group-with-ldap[2655675]: Skipping.
Oct 16 09:10:50 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:50,335 The conflicting address is held by maxbinderwmf: https://gitlab.wikimedia.org/maxbinderwmf
Oct 16 09:10:50 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:50,398 Creating gitlab user yliou
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:51,017 Failed to create user yliou with email address yliou@wikimedia.org: Email has already been taken
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2655675]: Skipping.
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:51,104 The conflicting address is held by yumingliou: https://gitlab.wikimedia.org/yumingliou
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:51,104 Failed to create 4 users.
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2655675]: 2026-10-16 09:10:51,104 Sync completed.
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:51,659 Collecting membership list of LDAP group ops
Oct 16 09:10:51 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:51,700 Collecting member list of Gitlab group repos/sre
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:52,288 There are 0 GitLab users to create.
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:52,288 There are 0 members to add to repos/sre.
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:52,288 There are 0 members to remove from repos/sre.
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:52,289 No work to do.
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669464]: 2026-10-16 09:10:52,289 Sync completed.
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:52,862 Collecting membership list of LDAP group wmde
Oct 16 09:10:52 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:52,912 Collecting member list of Gitlab group people/wmde
Oct 16 09:10:53 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:53,552 There are 0 GitLab users to create.
Oct 16 09:10:53 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:53,552 There are 0 members to add to people/wmde.
Oct 16 09:10:53 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:53,552 There are 0 members to remove from people/wmde.
Oct 16 09:10:53 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:53,553 No work to do.
Oct 16 09:10:53 gitlab1004 sync-gitlab-group-with-ldap[2669736]: 2026-10-16 09:10:53,553 Sync completed.
Oct 16 09:10:53 gitlab1004 systemd[1]: sync-gitlab-group-with-ldap.service: Deactivated successfully.
Oct 16 09:10:53 gitlab1004 systemd[1]: Finished sync-gitlab-group-with-ldap.service - Sync various GitLab groups with their LDAP groups.
Oct 16 09:10:53 gitlab1004 systemd[1]: sync-gitlab-group-with-ldap.service: Consumed 1.450s CPU time.

Fri, Oct 16, 9:11 AM · Collaboration-Services

ABran-WMF closed T441394: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100, a subtask of T423984: Add retries, error handling and metrics to sync-gitlab-group-with-ldap, as Resolved.

Fri, Oct 16, 9:11 AM · Collaboration-Services

ABran-WMF added a subtask for T423984: Add retries, error handling and metrics to sync-gitlab-group-with-ldap: T441394: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100.

Fri, Oct 16, 9:11 AM · Collaboration-Services

ABran-WMF added a parent task for T441394: SystemdUnitFailed - sync-gitlab-group-with-ldap.service on gitlab1004:9100: T423984: Add retries, error handling and metrics to sync-gitlab-group-with-ldap.

Fri, Oct 16, 9:11 AM · Collaboration-Services

Yesterday

ABran-WMF merged task T441237: ProbeDown - gitlab1004 into Restricted Task.

Thu, Oct 15, 9:26 AM · Collaboration-Services

Wed, Oct 14

ABran-WMF closed T441112: GerritHAProxyServiceUnavailable as Resolved.

caused by a maintenance done by @hashar

Wed, Oct 14, 10:19 AM · Collaboration-Services

ABran-WMF merged task T441111: GerritHAProxyBackendUnavailable into T441112: GerritHAProxyServiceUnavailable.

Wed, Oct 14, 10:19 AM · Collaboration-Services

ABran-WMF merged T441111: GerritHAProxyBackendUnavailable into T441112: GerritHAProxyServiceUnavailable.

Wed, Oct 14, 10:19 AM · Collaboration-Services

ABran-WMF closed T440888: Test Gerrit against the new LDAP server, a subtask of T331699: Migrate the r/w LDAP servers to Trixie and MDB storage, as Resolved.

Wed, Oct 14, 8:47 AM · Patch-For-Review, LDAP, Infrastructure-Foundations, SRE

ABran-WMF closed T440888: Test Gerrit against the new LDAP server as Resolved.

Gerrit works fine with the new LDAP replicas, I was able to get a successful authentication with my credentials with the current and new ldap.

Wed, Oct 14, 8:47 AM · Collaboration-Services, LDAP, Infrastructure-Foundations

Mon, Oct 12

ABran-WMF renamed T440888: Test Gerrit against the new LDAP server from Test Gerrit agains the new LDAP server to Test Gerrit against the new LDAP server.

Mon, Oct 12, 3:20 PM · Collaboration-Services, LDAP, Infrastructure-Foundations

ABran-WMF changed the status of T440888: Test Gerrit against the new LDAP server, a subtask of T331699: Migrate the r/w LDAP servers to Trixie and MDB storage, from Open to In Progress.

Mon, Oct 12, 3:07 PM · Patch-For-Review, LDAP, Infrastructure-Foundations, SRE

ABran-WMF changed the status of T440888: Test Gerrit against the new LDAP server from Open to In Progress.

Mon, Oct 12, 3:07 PM · Collaboration-Services, LDAP, Infrastructure-Foundations

ABran-WMF claimed T440888: Test Gerrit against the new LDAP server.

Mon, Oct 12, 3:06 PM · Collaboration-Services, LDAP, Infrastructure-Foundations

Fri, Oct 9

ABran-WMF updated the task description for T420184: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org.

Fri, Oct 9, 8:02 AM · Patch-For-Review, Infrastructure-Foundations, Puppet, Gerrit, Collaboration-Services

Wed, Oct 7

ABran-WMF changed the status of T396979: Implement service level indicator measurement for Gerrit from Stalled to Open.

Wed, Oct 7, 1:34 PM · SRE-SLO, Collaboration-Services

ABran-WMF added a comment to T435197: Research packages size on GitLab.

In T435197#12379484, @fkaelin wrote:

If there is an alert or other mechanism we can clean up at a regular cadence.

Wed, Oct 7, 12:27 PM · Collaboration-Services, Research

ABran-WMF closed T435197: Research packages size on GitLab, a subtask of T316553: Setup alerting for GitLab projects size limits, as Resolved.

Wed, Oct 7, 12:24 PM · Collaboration-Services, GitLab (Infrastructure)

ABran-WMF closed T435197: Research packages size on GitLab as Resolved.

I think we can consider this resolved.

Wed, Oct 7, 12:24 PM · Collaboration-Services, Research

ABran-WMF updated subscribers of T440177: Add the the ability to warm caches to Gerrit switchover cookbook.

thanks for the additional patch @Dzahn

Wed, Oct 7, 10:09 AM · Patch-For-Review, User-ABran-WMF, Gerrit, Collaboration-Services, Sustainability (Incident Followup), Incident Severity 1

Tue, Oct 6

ABran-WMF changed the status of T440177: Add the the ability to warm caches to Gerrit switchover cookbook from Open to In Progress.

@hashar the 3 patches above are what I had in mind, to allow us recovering from a failover quicker.

Tue, Oct 6, 2:34 PM · Patch-For-Review, User-ABran-WMF, Gerrit, Collaboration-Services, Sustainability (Incident Followup), Incident Severity 1

ABran-WMF changed the status of T440177: Add the the ability to warm caches to Gerrit switchover cookbook, a subtask of T439025: Gerrit failover, from Open to In Progress.

Tue, Oct 6, 2:34 PM · Gerrit, User-ABran-WMF, Patch-For-Review, Collaboration-Services

ABran-WMF changed the status of T440177: Add the the ability to warm caches to Gerrit switchover cookbook, a subtask of T439010: codfw power outage, from Open to In Progress.

Tue, Oct 6, 2:34 PM · Incident Severity 1, Wikimedia-Incident

Mon, Oct 5

ABran-WMF added a comment to T440177: Add the the ability to warm caches to Gerrit switchover cookbook.

With a bit more digging in Gerrit's doc, it seems that we can not build these indexes/caches at runtime.

Mon, Oct 5, 12:03 PM · Patch-For-Review, User-ABran-WMF, Gerrit, Collaboration-Services, Sustainability (Incident Followup), Incident Severity 1

ABran-WMF triaged T440177: Add the the ability to warm caches to Gerrit switchover cookbook as High priority.

Mon, Oct 5, 11:56 AM · Patch-For-Review, User-ABran-WMF, Gerrit, Collaboration-Services, Sustainability (Incident Followup), Incident Severity 1

ABran-WMF added a subtask for T439025: Gerrit failover: T440177: Add the the ability to warm caches to Gerrit switchover cookbook.

Mon, Oct 5, 11:55 AM · Gerrit, User-ABran-WMF, Patch-For-Review, Collaboration-Services

ABran-WMF added a parent task for T440177: Add the the ability to warm caches to Gerrit switchover cookbook: T439025: Gerrit failover.

Mon, Oct 5, 11:55 AM · Patch-For-Review, User-ABran-WMF, Gerrit, Collaboration-Services, Sustainability (Incident Followup), Incident Severity 1

ABran-WMF added projects to T440177: Add the the ability to warm caches to Gerrit switchover cookbook: Gerrit, User-ABran-WMF.

I started working on a failover cookbook that follows the process that was documented/improved and used.

Mon, Oct 5, 11:53 AM · Patch-For-Review, User-ABran-WMF, Gerrit, Collaboration-Services, Sustainability (Incident Followup), Incident Severity 1

ABran-WMF moved T418215: Update gerrit replication plugin with new metrics from Work in Progress to Backlog on the Collaboration-Services board.

This is tied to Gerrit (Gerrit 3.11)

Mon, Oct 5, 6:17 AM · Upstream, Gerrit (Gerrit 3.11), Release-Engineering-Team, Collaboration-Services

ABran-WMF closed T439209: Cleanup duplicated gerrit.service after systemctl edit --full as Resolved.

First run with puppet disabled on Gerrit primary:

arnaudb@gerrit2003:~ $ sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for gerrit2003.wikimedia.org
Info: Applying configuration version '(709725b03d) Arnaudb - gerrit: remove a unit copy left by systemctl edit --full'
Notice: /Stage[main]/Gerrit/Exec[systemctl revert gerrit.service]/returns: executed successfully
Notice: Applied catalog in 26.34 seconds
arnaudb@gerrit2003:~ $ systemctl show -p FragmentPath gerrit.service
FragmentPath=/lib/systemd/system/gerrit.service
arnaudb@gerrit2003:~ $ systemctl status gerrit
● gerrit.service - Gerrit code review tool
     Loaded: loaded (/lib/systemd/system/gerrit.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-09-23 20:00:32 UTC; 1 week 4 days ago

Mon, Oct 5, 6:11 AM · Gerrit, Collaboration-Services

ABran-WMF closed T439209: Cleanup duplicated gerrit.service after systemctl edit --full, a subtask of T439025: Gerrit failover, as Resolved.

Mon, Oct 5, 6:11 AM · Gerrit, User-ABran-WMF, Patch-For-Review, Collaboration-Services

Thu, Oct 1

ABran-WMF changed the status of T436657: Move aphlict to k8s from Open to In Progress.

Thu, Oct 1, 9:38 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF updated the task description for T425441: gitlab behind CDN.

Thu, Oct 1, 8:01 AM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF updated the task description for T437216: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org.

Thu, Oct 1, 7:24 AM · Machine-Learning-Team (Q1 FY2026-27), SecTeam-Processed, Product Safety and Integrity, Research, Security-Team, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

Tue, Sep 29

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Tue, Sep 29, 2:05 PM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Mon, Sep 28

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Mon, Sep 28, 12:06 PM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Mon, Sep 28, 9:15 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Fri, Sep 25

ABran-WMF added projects to T439025: Gerrit failover: User-ABran-WMF, Gerrit.

Fri, Sep 25, 9:09 AM · Gerrit, User-ABran-WMF, Patch-For-Review, Collaboration-Services

ABran-WMF changed the status of T439209: Cleanup duplicated gerrit.service after systemctl edit --full from Open to In Progress.

Fri, Sep 25, 9:01 AM · Gerrit, Collaboration-Services

ABran-WMF changed the status of T439209: Cleanup duplicated gerrit.service after systemctl edit --full, a subtask of T439025: Gerrit failover, from Open to In Progress.

Fri, Sep 25, 9:01 AM · Gerrit, User-ABran-WMF, Patch-For-Review, Collaboration-Services

ABran-WMF added a comment to T439209: Cleanup duplicated gerrit.service after systemctl edit --full.

I also updated the doc accordingly : https://wikitech.wikimedia.org/wiki/Gerrit/Operations#Failover_(in_case_primary_host_is_unavailable)

Fri, Sep 25, 9:00 AM · Gerrit, Collaboration-Services

ABran-WMF created T439209: Cleanup duplicated gerrit.service after systemctl edit --full.

Fri, Sep 25, 7:52 AM · Gerrit, Collaboration-Services

ABran-WMF closed T439104: gerrit: rotate spare and replica httpd logs, a subtask of T439039: GerritDiskSpaceExhaustionIncoming, as Resolved.

Fri, Sep 25, 7:30 AM · Collaboration-Services

ABran-WMF closed T439104: gerrit: rotate spare and replica httpd logs, a subtask of T425667: Investigate Gerrit root disk usage and logging, as Resolved.

Fri, Sep 25, 7:30 AM · Gerrit, Collaboration-Services

ABran-WMF closed T439104: gerrit: rotate spare and replica httpd logs as Resolved.

Fri, Sep 25, 7:30 AM · Collaboration-Services

Thu, Sep 24

ABran-WMF added a subtask for T422070: GitlabPackagePullerFailedOnRun - apt-staging2001:9100: T439056: Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare.

Thu, Sep 24, 10:19 AM · Collaboration-Services

ABran-WMF added a parent task for T439056: Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare: T422070: GitlabPackagePullerFailedOnRun - apt-staging2001:9100.

Thu, Sep 24, 10:19 AM · Collaboration-Services

ABran-WMF updated the task description for T439056: Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare.

Thu, Sep 24, 10:18 AM · Collaboration-Services

ABran-WMF updated the task description for T439056: Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare.

Thu, Sep 24, 10:18 AM · Collaboration-Services

ABran-WMF renamed T439056: Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare from GitlabPackagePullerFailedOnPrepare to Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare.

Thu, Sep 24, 10:17 AM · Collaboration-Services

ABran-WMF triaged T439056: Add a runbook and a dashboard to GitlabPackagePullerFailedOnPrepare as Low priority.

it seems its silence expired:

Thu, Sep 24, 10:16 AM · Collaboration-Services

ABran-WMF claimed T439104: gerrit: rotate spare and replica httpd logs.

Thu, Sep 24, 8:40 AM · Collaboration-Services

ABran-WMF moved T439104: gerrit: rotate spare and replica httpd logs from Incoming to Awaiting Input on the Collaboration-Services board.

Thu, Sep 24, 8:39 AM · Collaboration-Services

ABran-WMF renamed T439104: gerrit: rotate spare and replica httpd logs from gerrit: spare and replica httpd logs are not rotated to gerrit: rotate spare and replica httpd logs.

Thu, Sep 24, 8:32 AM · Collaboration-Services

ABran-WMF moved T439039: GerritDiskSpaceExhaustionIncoming from Incoming to Awaiting Input on the Collaboration-Services board.

Thu, Sep 24, 8:31 AM · Collaboration-Services

ABran-WMF added a comment to T439039: GerritDiskSpaceExhaustionIncoming.

I updated the threshold so it is less likely to alert on false positives

Thu, Sep 24, 8:30 AM · Collaboration-Services

ABran-WMF triaged T439104: gerrit: rotate spare and replica httpd logs as Medium priority.

Thu, Sep 24, 8:28 AM · Collaboration-Services

ABran-WMF added a subtask for T425667: Investigate Gerrit root disk usage and logging: T439104: gerrit: rotate spare and replica httpd logs.

Thu, Sep 24, 8:28 AM · Gerrit, Collaboration-Services

ABran-WMF added a parent task for T439104: gerrit: rotate spare and replica httpd logs: T425667: Investigate Gerrit root disk usage and logging.

Thu, Sep 24, 8:28 AM · Collaboration-Services

ABran-WMF added a subtask for T439039: GerritDiskSpaceExhaustionIncoming: T439104: gerrit: rotate spare and replica httpd logs.

Thu, Sep 24, 8:24 AM · Collaboration-Services

ABran-WMF added a parent task for T439104: gerrit: rotate spare and replica httpd logs: T439039: GerritDiskSpaceExhaustionIncoming.

Thu, Sep 24, 8:24 AM · Collaboration-Services

ABran-WMF created T439104: gerrit: rotate spare and replica httpd logs.

Thu, Sep 24, 8:24 AM · Collaboration-Services

ABran-WMF claimed T439039: GerritDiskSpaceExhaustionIncoming.

Thu, Sep 24, 8:11 AM · Collaboration-Services

Wed, Sep 23

ABran-WMF updated subscribers of T436657: Move aphlict to k8s.

The minor-vs-patch question from the Phase 0 section above is resolved: @JMeybohm confirmed on 1338751 that a patch bump of 1.16.x is fine as long as it stays backward compatible. The patches have been updated accordingly.

Wed, Sep 23, 9:11 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Tue, Sep 22

ABran-WMF added a comment to T436657: Move aphlict to k8s.

the first version of Aphlict's Docker image has been published:
2026-09-22 15:10:22,893 Target 'production' image published to 'docker-registry.wikimedia.org/repos/phabricator/aphlict:2026-09-22-150917@sha256:efa11856126ea3bf84501b3c13d3fd1349130dca7178b24dcde2d7edea9b239b'

Tue, Sep 22, 3:13 PM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF added a comment to T438786: SystemdUnitFailed - zuul1005.

Alert silenced for 24 hours with 31530d81-c143-46f5-bb13-fc8a2ef985d4

Tue, Sep 22, 12:36 PM · Collaboration-Services

ABran-WMF merged T438839: SystemdUnitFailed into T438786: SystemdUnitFailed - zuul1005.

Tue, Sep 22, 12:36 PM · Collaboration-Services

ABran-WMF merged task T438839: SystemdUnitFailed into T438786: SystemdUnitFailed - zuul1005.

Tue, Sep 22, 12:36 PM · Collaboration-Services

ABran-WMF added a project to T182759: Add Prometheus exporter to Jenkins instances: User-ABran-WMF.

Tue, Sep 22, 8:34 AM · User-ABran-WMF, Release-Engineering-Team (Seen), observability, Continuous-Integration-Infrastructure, Goal, SRE

Wed, Sep 16

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Wed, Sep 16, 11:22 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Sep 16 2026

ABran-WMF changed the status of T438117: Add aux-k8s-worker100(6-9) and aux-k8s-worker200(6-9) to the aux-k8s cluster, a subtask of T378742: aux-k8s: eqiad expansion, codfw creation, & future hopes and dreams, from Open to In Progress.

Sep 16 2026, 9:10 AM · Infrastructure-Foundations, SRE, Kubernetes, Epic

ABran-WMF changed the status of T438117: Add aux-k8s-worker100(6-9) and aux-k8s-worker200(6-9) to the aux-k8s cluster from Open to In Progress.

it seems the nodes are already up and running:

arnaudb@cumin1003:~ $ sudo cumin  'aux-k8s-ctrl1002.eqiad.wmnet' "kubectl get nodes |grep -iE 'worker100(6|7|8|9)'; kubectl get pods --all-namespaces -o wide | grep -iE 'worker100(6|7|8|9)'"
1 hosts will be targeted:
aux-k8s-ctrl1002.eqiad.wmnet
OK to proceed on 1 hosts? Enter the number of affected hosts to confirm or "q" to quit: 1
----- OUTPUT for command #1: 'kubectl get node...ker100(6|7|8|9)'' -----                                                                                                                                                                
aux-k8s-worker1006.eqiad.wmnet   Ready    <none>          159d   v1.31.14                                                                                                                                                               
aux-k8s-worker1007.eqiad.wmnet   Ready    <none>          159d   v1.31.14                                                                                                                                                               
aux-k8s-worker1008.eqiad.wmnet   Ready    <none>          159d   v1.31.14                                                                                                                                                               
aux-k8s-worker1009.eqiad.wmnet   Ready    <none>          159d   v1.31.14                                                                                                                                                               
cert-manager        cert-manager-566d74fbf7-h8t6p                                     1/1     Running   0                 15d    10.67.86.194   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
cert-manager        cert-manager-cainjector-6894f7598c-wx4xb                          1/1     Running   0                 15d    10.67.86.205   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
cert-manager        cert-manager-webhook-5c8975f48d-7n4p9                             1/1     Running   0                 15d    10.67.86.193   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
cert-manager        cfssl-issuer-6cfd474fdb-qmrfc                                     1/1     Running   0                 15d    10.67.86.200   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
etherpad            etherpad-main-7fcf7c9bc9-xk7pd                                    2/2     Running   0                 19h    10.67.82.127   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
istio-system        istio-ingressgateway-cwgpb                                        1/1     Running   5 (15d ago)       159d   10.67.82.102   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
istio-system        istio-ingressgateway-pw54b                                        1/1     Running   5 (15d ago)       159d   10.67.82.199   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
istio-system        istio-ingressgateway-t2txp                                        1/1     Running   5 (15d ago)       159d   10.67.84.173   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
istio-system        istio-ingressgateway-tkldt                                        1/1     Running   5 (15d ago)       159d   10.67.86.203   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
istio-system        istiod-84cf8cf684-42cwk                                           1/1     Running   0                 15d    10.67.86.220   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
istio-system        istiod-84cf8cf684-k589q                                           1/1     Running   0                 15d    10.67.82.198   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
jaeger              main-jaeger-collector-587578df84-jsgvn                            1/1     Running   0                 94m    10.67.82.71    aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
jaeger              main-jaeger-query-d6d5cd86b-gm9mk                                 2/2     Running   0                 94m    10.67.82.209   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
jaeger              main-jaeger-query-d6d5cd86b-gnplb                                 2/2     Running   0                 94m    10.67.82.68    aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-jumbo-eqiad-to-test-eqiad-75b47c5895-fwl66      1/1     Running   0                 15d    10.67.86.219   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-codfw-to-main-eqiad-d597674fd-8z5sk        1/1     Running   1 (7d18h ago)     15d    10.67.84.174   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-codfw-to-main-eqiad-d597674fd-pq4c6        1/1     Running   1 (7d18h ago)     15d    10.67.82.193   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-codfw-to-main-eqiad-d597674fd-w665n        1/1     Running   1 (7d18h ago)     15d    10.67.86.221   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-jumbo-eqiad-749cd95955-fmv6d      1/1     Running   0                 15d    10.67.84.177   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-jumbo-eqiad-749cd95955-k92df      1/1     Running   0                 15d    10.67.86.212   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-jumbo-eqiad-749cd95955-pmfhl      1/1     Running   0                 15d    10.67.86.210   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-jumbo-eqiad-749cd95955-r54r5      1/1     Running   0                 15d    10.67.82.203   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-main-codfw-7fcd8d547f-gqtr4       1/1     Running   1 (7d18h ago)     15d    10.67.84.176   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-main-codfw-7fcd8d547f-x9zpn       1/1     Running   0                 15d    10.67.82.195   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-main-codfw-7fcd8d547f-xx58c       1/1     Running   1 (7d18h ago)     15d    10.67.84.175   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-main-codfw-7fcd8d547f-zgmfm       1/1     Running   1 (7d18h ago)     15d    10.67.86.213   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kube-system         calico-node-745qm                                                 1/1     Running   5 (15d ago)       159d   10.64.183.2    aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kube-system         calico-node-mq2p8                                                 1/1     Running   6 (15d ago)       159d   10.64.187.2    aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kube-system         calico-node-qrrhg                                                 1/1     Running   7 (15d ago)       159d   10.64.0.80     aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
kube-system         calico-node-sj4rb                                                 1/1     Running   7 (15d ago)       159d   10.64.16.132   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
kube-system         ceph-csi-rbd-nodeplugin-cmnlh                                     2/2     Running   6 (15d ago)       78d    10.64.16.132   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
kube-system         ceph-csi-rbd-nodeplugin-lks7h                                     2/2     Running   6 (15d ago)       78d    10.64.187.2    aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kube-system         ceph-csi-rbd-nodeplugin-nmds2                                     2/2     Running   6 (15d ago)       78d    10.64.0.80     aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
kube-system         ceph-csi-rbd-nodeplugin-ts6tj                                     2/2     Running   6 (15d ago)       78d    10.64.183.2    aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
kube-system         ceph-csi-rbd-provisioner-6b98c7f56c-q8d9w                         6/6     Running   0                 15d    10.67.84.180   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kube-system         coredns-7fbbb48bd6-frpwv                                          1/1     Running   0                 15d    10.67.82.210   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
kube-system         coredns-7fbbb48bd6-gtpxm                                          1/1     Running   0                 15d    10.67.84.168   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
kube-system         coredns-7fbbb48bd6-wztr5                                          1/1     Running   0                 15d    10.67.86.211   aux-k8s-worker1006.eqiad.wmnet   <none>           <none>                                
miscweb             miscweb-os-reports-584bb6879c-dkgdt                               4/4     Running   0                 8d     10.67.84.183   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
miscweb             miscweb-os-reports-584bb6879c-rdnkm                               4/4     Running   0                 8d     10.67.82.122   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
miscweb             miscweb-wmf-navigator-5d5f9444b6-2kdm7                            5/5     Running   300 (3m32s ago)   8d     10.67.82.208   aux-k8s-worker1008.eqiad.wmnet   <none>           <none>                                
miscweb             miscweb-wmf-navigator-5d5f9444b6-p5v24                            5/5     Running   308 (57m ago)     8d     10.67.82.116   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
redioscope          redioscope-main-6f7948b6b5-rm256                                  1/1     Running   0                 15d    10.67.84.144   aux-k8s-worker1007.eqiad.wmnet   <none>           <none>                                
zarcillo            zarcillo-main-84d6dcdb8b-pcmjx                                    2/2     Running   0                 46h    10.67.82.121   aux-k8s-worker1009.eqiad.wmnet   <none>           <none>                                
================                                                                                                                                                                                                                        
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00,  1.52s/hosts]
FAIL |                                                                                                                                                                                                  |   0% (0/1) [00:01<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command #1: 'kubectl get node...ker100(6|7|8|9)''.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
arnaudb@cumin1003:~ $ sudo cumin  'aux-k8s-ctrl2002.codfw.wmnet' "kubectl get nodes | grep worker200; kubectl get pods --all-namespaces -o wide | grep -iE 'worker200(6|7|8|9)'"
1 hosts will be targeted:
aux-k8s-ctrl2002.codfw.wmnet
OK to proceed on 1 hosts? Enter the number of affected hosts to confirm or "q" to quit: 1
----- OUTPUT for command #1: 'kubectl get node...ker200(6|7|8|9)'' -----                                                                                                                                                                
aux-k8s-worker2002.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2003.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2004.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2005.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2006.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2007.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2008.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
aux-k8s-worker2009.codfw.wmnet   Ready    <none>          161d   v1.31.14                                                                                                                                                               
cert-manager        cert-manager-566d74fbf7-94x9v                                 1/1     Running   0                 15d    10.194.81.199   aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
istio-system        istio-ingressgateway-48bzp                                    1/1     Running   5 (15d ago)       161d   10.194.84.240   aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
istio-system        istio-ingressgateway-k6hms                                    1/1     Running   5 (15d ago)       161d   10.194.81.198   aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
istio-system        istio-ingressgateway-rpsm2                                    1/1     Running   5 (15d ago)       161d   10.194.87.240   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
istio-system        istio-ingressgateway-svrq2                                    1/1     Running   5 (15d ago)       161d   10.194.87.44    aux-k8s-worker2009.codfw.wmnet   <none>           <none>                                   
jaeger              main-jaeger-collector-587578df84-87km6                        1/1     Running   0                 15d    10.194.87.241   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
jaeger              main-jaeger-query-d6d5cd86b-2wgxt                             2/2     Running   0                 89m    10.194.87.45    aux-k8s-worker2009.codfw.wmnet   <none>           <none>                                   
jaeger              main-jaeger-query-d6d5cd86b-rhtlb                             2/2     Running   0                 89m    10.194.87.249   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
kafka-mirrormaker   kafka-mirrormaker-main-codfw-to-main-eqiad-d597674fd-95qpd    1/1     Running   1 (7d18h ago)     15d    10.194.84.241   aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
kafka-mirrormaker   kafka-mirrormaker-main-codfw-to-main-eqiad-d597674fd-f4p64    1/1     Running   0                 15d    10.194.81.192   aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-main-codfw-7fcd8d547f-dfjk2   1/1     Running   1 (7d18h ago)     15d    10.194.84.242   aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
kafka-mirrormaker   kafka-mirrormaker-main-eqiad-to-main-codfw-7fcd8d547f-lb6wx   1/1     Running   0                 15d    10.194.81.196   aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
kafka-ui            kafka-ui-production-5c9c755b9b-7ndbg                          3/3     Running   0                 15d    10.194.81.205   aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
kube-system         calico-kube-controllers-86488dfbf5-wlmmn                      1/1     Running   0                 15d    10.194.84.243   aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
kube-system         calico-node-5n7gm                                             1/1     Running   6 (15d ago)       161d   10.192.43.14    aux-k8s-worker2009.codfw.wmnet   <none>           <none>                                   
kube-system         calico-node-68shf                                             1/1     Running   6 (15d ago)       161d   10.192.4.23     aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
kube-system         calico-node-7jrtk                                             1/1     Running   5 (15d ago)       161d   10.192.6.21     aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
kube-system         calico-node-fksx8                                             1/1     Running   5 (15d ago)       161d   10.192.12.36    aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
kube-system         calico-typha-7c5b67b766-dfkk5                                 1/1     Running   0                 15d    10.192.12.36    aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
kube-system         coredns-7fbbb48bd6-bsrh8                                      1/1     Running   0                 15d    10.194.84.244   aux-k8s-worker2007.codfw.wmnet   <none>           <none>                                   
kube-system         coredns-7fbbb48bd6-kfbmj                                      1/1     Running   0                 15d    10.194.87.243   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
kube-system         coredns-7fbbb48bd6-q2wx9                                      1/1     Running   0                 15d    10.194.81.202   aux-k8s-worker2006.codfw.wmnet   <none>           <none>                                   
kube-system         helm-state-metrics-75869c4854-jl7rt                           1/1     Running   0                 15d    10.194.87.244   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
kube-system         kube-state-metrics-8676cdb6b5-qns74                           1/1     Running   0                 15d    10.194.87.245   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
miscweb             miscweb-os-reports-798995dd8f-fh682                           4/4     Running   0                 15d    10.194.87.246   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
miscweb             miscweb-wmf-navigator-775b7dd969-f892x                        5/5     Running   565 (3m51s ago)   15d    10.194.87.247   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
sophroid            sophroid-main-5f49c79485-rdh8f                                1/1     Running   0                 15d    10.194.87.248   aux-k8s-worker2008.codfw.wmnet   <none>           <none>                                   
================                                                                                                                                                                                                                        
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00,  1.63s/hosts]
FAIL |                                                                                                                                                                                                  |   0% (0/1) [00:01<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command #1: 'kubectl get node...ker200(6|7|8|9)''.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Sep 16 2026, 9:10 AM · Collaboration-Services, Infrastructure-Foundations, Kubernetes

Sep 15 2026

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Sep 15 2026, 9:13 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF added a subtask for T436657: Move aphlict to k8s: T340169: consider moving aphlict admin port to https / envoy.

Sep 15 2026, 8:26 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF added a parent task for T340169: consider moving aphlict admin port to https / envoy: T436657: Move aphlict to k8s.

Sep 15 2026, 8:26 AM · Collaboration-Services, Phabricator

Sep 14 2026

ABran-WMF added a comment to T425441: gitlab behind CDN.

Following up on:

In T425441#12285450, @ABran-WMF wrote:

thanks for mentioning it @jnuche
I sent 2 patches, one that will enable websockets on Envoy, the other to map them on our backends. That should fix the issue you're facing.

Sep 14 2026, 2:18 PM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Sep 14 2026, 10:40 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Sep 11 2026

ABran-WMF added a comment to T435509: Deploy etherpad-new to aux-k8s cluster.

Heads up: the mesh-side websocket support you deferred long-polling for is up for review as a four-change stack, starting at 1338959 with the feature itself in 1338751, then 1338960 and 1338752. mesh.websockets.enabled is off by default and inert for every existing chart, and a mesh.restricted_listeners entry opts in separately from mesh.public_port.

Sep 11 2026, 9:00 AM · Patch-For-Review, Collaboration-Services, Wikimedia-Etherpad

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Sep 11 2026, 8:53 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Sep 11 2026, 8:43 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Sep 9 2026

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Sep 9 2026, 10:27 AM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Sep 8 2026

ABran-WMF updated the task description for T436657: Move aphlict to k8s.

Sep 8 2026, 12:37 PM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

Sep 7 2026

ABran-WMF added a project to T436049: Consider disabling paging for lists.wikimedia.org: User-ABran-WMF.

Sep 7 2026, 1:41 PM · User-ABran-WMF, Traffic, Sustainability (Incident Followup), Collaboration-Services, SRE Observability, Wikimedia-Mailing-lists

ABran-WMF triaged T436657: Move aphlict to k8s as Medium priority.

Sep 7 2026, 1:40 PM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF moved T436657: Move aphlict to k8s from Incoming to Up Next on the Collaboration-Services board.

Sep 7 2026, 1:40 PM · Patch-For-Review, Release-Engineering-Team, Phabricator, Collaboration-Services, User-ABran-WMF

ABran-WMF closed T437189: GitLab Replicas should not page, a subtask of T425441: gitlab behind CDN, as Resolved.

Sep 7 2026, 12:56 PM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF closed T437189: GitLab Replicas should not page as Resolved.

Sep 7 2026, 12:56 PM · GitLab, Collaboration-Services

ABran-WMF created T437216: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org.

Sep 7 2026, 12:24 PM · Machine-Learning-Team (Q1 FY2026-27), SecTeam-Processed, Product Safety and Integrity, Research, Security-Team, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF added a comment to T437189: GitLab Replicas should not page.

The primary issue here has been fixed with the exclusion of Gitlab replicas from the ATS alert that paged. Gitlab's primary instance has also been excluded from future upgrade related alerts.

Sep 7 2026, 9:50 AM · GitLab, Collaboration-Services

ABran-WMF triaged T437189: GitLab Replicas should not page as High priority.

Sep 7 2026, 8:33 AM · GitLab, Collaboration-Services

ABran-WMF added a comment to T425441: gitlab behind CDN.

Good catch, and unlike the dbt row that one is a genuine call across the edge. It passes, because Poetry sets its own User-Agent.

Sep 7 2026, 8:29 AM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF added a comment to T425441: gitlab behind CDN.

In T425441#12293630, @brouberol wrote:
It's interesting to see
an-worker1200.eqiad.wmnet   Go-http-client/1.1 -> 401   python-requests/2.32.3 -> 401
in the list of affected clients. I'm not aware of any go binary being executed by the data engineering stack, so that might be something installed by the system itself?

When it comes to
src/main/java/org/apache/hadoop/fs/http/CopiedAbstractHttpFileSystem.java:61, src/main/java/org/apache/hadoop/fs/http/WmfHttpsFileSystem.java:40
we should stop seeing these issues, as I've reconfigured all DE jobs to use the gitlab.discovery.wmnet:8443 https endpoint. If that's not the case, I might have missed something.

The dbt case is interesting, as this is a python build system fetching/cloning a package at build time. I will investigate whether we can tune the user agent of said build system.

Sep 7 2026, 8:06 AM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF added a comment to T425441: gitlab behind CDN.

Three corrections to T425441#12288782. Line numbers below are in modules/profile/templates/cache/haproxy/tls_terminator.cfg.erb.

Sep 7 2026, 8:02 AM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

Sep 4 2026

ABran-WMF added a comment to T425441: gitlab behind CDN.

edit: upon further review, the table bellow has a lots of error, please refer to T437216: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org for that matter.

Sep 4 2026, 9:23 AM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

Sep 3 2026

ABran-WMF added a comment to T425441: gitlab behind CDN.

Gitlab primary instance is back to where it was this morning to simplify debugging the missing user-agent issue.

Sep 3 2026, 2:20 PM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF updated subscribers of T425441: gitlab behind CDN.

Another issue was mentioned by @trueg. As discussed in T435292, highlighting T400119: Block traffic from user-agents not honoring our policy → we ask for requests to the CDN to have a user agent header, that might create some issue on code/CI not complying with that.

Sep 3 2026, 1:08 PM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF added a comment to T425441: gitlab behind CDN.

thanks for mentioning it @jnuche
I sent 2 patches, one that will enable websockets on Envoy, the other to map them on our backends. That should fix the issue you're facing.

Sep 3 2026, 12:18 PM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF added a subtask for T425441: gitlab behind CDN: T436902: GitLab agents failing to connect.

Sep 3 2026, 12:17 PM · Patch-For-Review, Release-Engineering-Team (Radar), GitLab, Collaboration-Services

ABran-WMF added a parent task for T436902: GitLab agents failing to connect: T425441: gitlab behind CDN.

Sep 3 2026, 12:17 PM · Release-Engineering-Team (Doing 😎), GitLab, Essential-Work, Catalyst (Luka Ijo Pimeja Jan)

ABran-WMF (arnaudb)
SRE

Projects (8)
View All

Calendar

Today

Tomorrow

Sunday

User Details

Recent Activity
View All

Today

Yesterday

Wed, Oct 14

Mon, Oct 12

Fri, Oct 9

Wed, Oct 7

Tue, Oct 6

Mon, Oct 5

Thu, Oct 1

Tue, Sep 29

Mon, Sep 28

Fri, Sep 25

Thu, Sep 24

Wed, Sep 23

Tue, Sep 22

Wed, Sep 16

Sep 16 2026

Sep 15 2026

Sep 14 2026

Sep 11 2026

Sep 9 2026

Sep 8 2026

Sep 7 2026

Sep 4 2026

Sep 3 2026

ABran-WMF (arnaudb)SRE

Projects (8)View All

Calendar

Today

Tomorrow

Sunday

User Details

Recent ActivityView All

Today

Yesterday

Wed, Oct 14

Mon, Oct 12

Fri, Oct 9

Wed, Oct 7

Tue, Oct 6

Mon, Oct 5

Thu, Oct 1

Tue, Sep 29

Mon, Sep 28

Fri, Sep 25

Thu, Sep 24

Wed, Sep 23

Tue, Sep 22

Wed, Sep 16

Sep 16 2026

Sep 15 2026

Sep 14 2026

Sep 11 2026

Sep 9 2026

Sep 8 2026

Sep 7 2026

Sep 4 2026

Sep 3 2026

ABran-WMF (arnaudb)
SRE

Projects (8)
View All

Recent Activity
View All