Page MenuHomePhabricator

LSobanski (Lukasz Sobanski)
Woo$

Today

  • No visible events.

Tomorrow

  • No visible events.

Tuesday

  • No visible events.

User Details

User Since
Aug 31 2020, 5:40 PM (283 w, 5 d)
Availability
Available
LDAP User
LSobanski
MediaWiki User
LSobanski (WMF) [ Global Accounts ]

Recent Activity

Mon, Feb 2

LSobanski added a comment to T416062: Grant Access to bitu-account-managers(?) for reedy.

Approved.

Mon, Feb 2, 10:11 AM · SRE, LDAP-Access-Requests

Thu, Jan 22

LSobanski added a project to T399158: Alert in need of triage: OsmSynchronisationLag (instance maps-test2001:9100): Infrastructure-Foundations.
Thu, Jan 22, 10:44 AM · Infrastructure-Foundations, SRE, Maps, sre-alert-triage

Tue, Jan 20

LSobanski moved T409137: lists.wikimedia.org subscription email rejected by DKIM from Incoming to Backlog on the collaboration-services board.
Tue, Jan 20, 9:35 AM · collaboration-services, Wikimedia-Mailing-lists, SRE, Infrastructure-Foundations

Mon, Jan 19

LSobanski moved T413781: gerrit: wrong network for public IPV4 and IPV6 from Incoming to Backlog on the collaboration-services board.
Mon, Jan 19, 4:20 PM · collaboration-services, Infrastructure-Foundations, Puppet-Core
LSobanski removed a project from T413179: deployment charts: automate testing on staging: collaboration-services.
Mon, Jan 19, 4:17 PM · ServiceOps-Services-Oids, Kubernetes, ServiceOps new, Developer Productivity
LSobanski moved T413871: Make sure GitLab does not exceed apus object storage quotas from Incoming to Backlog on the collaboration-services board.
Mon, Jan 19, 4:16 PM · collaboration-services, GitLab (Infrastructure)
LSobanski moved T414407: Gerrit topology evolution branch pick-up from Incoming to Backlog on the collaboration-services board.
Mon, Jan 19, 4:13 PM · Gerrit, collaboration-services
LSobanski triaged T414407: Gerrit topology evolution branch pick-up as Low priority.
Mon, Jan 19, 4:13 PM · Gerrit, collaboration-services
LSobanski moved T414098: Move https://status.wikimedia.org/ away from rackspace from Incoming to Consultation on the collaboration-services board.
Mon, Jan 19, 4:13 PM · collaboration-services, Patch-For-Review, SRE Observability, cloud-services-team
LSobanski reopened T414940: Handle httpd log surplus coming from Liberica, a subtask of T411895: gerrit behind CDN, as Open.
Mon, Jan 19, 4:12 PM · Patch-For-Review, Gerrit, collaboration-services
LSobanski reopened T414940: Handle httpd log surplus coming from Liberica as "Open".
Mon, Jan 19, 4:12 PM · Gerrit, collaboration-services
LSobanski assigned T414968: Alert in need of triage: PuppetConstantChange (instance cloudidp2001-dev:9100) to SLyngshede-WMF.
Mon, Jan 19, 3:50 PM · Infrastructure-Foundations, sre-alert-triage
LSobanski updated subscribers of T414486: Upgrade AUX clusters to kubernetes 1.31.

Unless anything unexpected happens, @elukey will handle this towards the end of the quarter.

Mon, Jan 19, 3:42 PM · Infrastructure-Foundations, Kubernetes, Prod-Kubernetes
LSobanski triaged T414486: Upgrade AUX clusters to kubernetes 1.31 as Medium priority.
Mon, Jan 19, 3:41 PM · Infrastructure-Foundations, Kubernetes, Prod-Kubernetes
LSobanski created T414971: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100).
Mon, Jan 19, 1:50 PM · Machine-Learning-Team, sre-alert-triage
LSobanski created T414970: Alert in need of triage: KubernetesAPIErrorRate.
Mon, Jan 19, 1:47 PM · Data-Platform-SRE (2026.01.23 - 2026.02.13), sre-alert-triage
LSobanski created T414969: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100).
Mon, Jan 19, 1:47 PM · Machine-Learning-Team, sre-alert-triage
LSobanski created T414968: Alert in need of triage: PuppetConstantChange (instance cloudidp2001-dev:9100).
Mon, Jan 19, 1:46 PM · Infrastructure-Foundations, sre-alert-triage
LSobanski added a comment to T412793: Rotate statuspage API keys.

I got a reminder about this today so updating the task as well.

Mon, Jan 19, 11:44 AM · SRE Observability

Tue, Jan 13

LSobanski created T414413: Alert in need of triage: KubernetesAPIErrorRate.
Tue, Jan 13, 9:34 AM · Data-Platform-SRE (2026.01.23 - 2026.02.13), sre-alert-triage

Mon, Jan 12

LSobanski edited projects for T414098: Move https://status.wikimedia.org/ away from rackspace, added: SRE Observability; removed Infrastructure-Foundations.
Mon, Jan 12, 3:36 PM · collaboration-services, Patch-For-Review, SRE Observability, cloud-services-team

Jan 8 2026

LSobanski added a comment to T411146: Application Security Review Request: Wikipedia 25 microsite.

@Catrope just to to let you know the planned official rollout of this website is Jan 15.

Jan 8 2026, 11:19 AM · Security, SecTeam-Processed, Security-Team, secscrum, Application Security Reviews, PES1.3.3 WP25 Easter Eggs
LSobanski set Security to security-bug on T411146: Application Security Review Request: Wikipedia 25 microsite.
Jan 8 2026, 11:18 AM · Security, SecTeam-Processed, Security-Team, secscrum, Application Security Reviews, PES1.3.3 WP25 Easter Eggs

Jan 5 2026

LSobanski triaged T413781: gerrit: wrong network for public IPV4 and IPV6 as Low priority.
Jan 5 2026, 3:29 PM · collaboration-services, Infrastructure-Foundations, Puppet-Core
LSobanski assigned T413193: git::clone can fail to checkout its remote branch, leading to unrecoverable failure to jhathaway.
Jan 5 2026, 3:28 PM · Infrastructure-Foundations, SRE
LSobanski assigned T413181: asw1-b12-drmrs stopped reporting metrics to ayounsi.
Jan 5 2026, 3:24 PM · Infrastructure-Foundations, netops
LSobanski triaged T412078: Alert in need of triage: SmartNotHealthy (instance sretest2006:9100) as Low priority.
Jan 5 2026, 3:24 PM · SRE, ops-codfw, DC-Ops, Infrastructure-Foundations, sre-alert-triage
LSobanski triaged T412826: Provide an official Docker image for CAS-SSO as Low priority.
Jan 5 2026, 3:23 PM · Infrastructure-Foundations, CAS-SSO
LSobanski closed T413430: SystemdUnitFailed as Resolved.

Fired two weeks ago and no longer active. Resolving.

Jan 5 2026, 8:44 AM · collaboration-services
LSobanski added a comment to T413745: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudcumin2001:9100).

Also for eqiad.

Jan 5 2026, 8:41 AM · Observability-Alerting, sre-alert-triage
LSobanski created T413745: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudcumin2001:9100).
Jan 5 2026, 8:40 AM · Observability-Alerting, sre-alert-triage
LSobanski created T413744: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cloudidp2001-dev:9100).
Jan 5 2026, 8:40 AM · Observability-Alerting, sre-alert-triage
LSobanski created T413743: Alert in need of triage: nrpe_Check_whether_ferm_is_active_by_checking_the_default_input_chain (instance cumin2002:9100).
Jan 5 2026, 8:40 AM · Observability-Alerting, sre-alert-triage
LSobanski created T413742: Alert in need of triage: HDFS topology check (instance an-master1003).
Jan 5 2026, 8:39 AM · Essential-Work, Data-Platform-SRE (2026.01.05 - 2026.01.23), sre-alert-triage

Dec 19 2025

LSobanski updated subscribers of T412975: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd").

@dancy do you have any clues as to the need for this MX server? Also cc'ing @taavi as it was suggested to me that you may have knowledge on the topic. Knowing this would help to fix it the right way.

Dec 19 2025, 4:32 PM · Patch-For-Review, collaboration-services, Beta-Cluster-Infrastructure

Dec 18 2025

LSobanski triaged T412780: Use encrypted rsync for Gerrit as Medium priority.
Dec 18 2025, 11:16 AM · Gerrit, collaboration-services

Dec 16 2025

LSobanski created T412793: Rotate statuspage API keys.
Dec 16 2025, 11:45 AM · SRE Observability
LSobanski created T412789: Alert in need of triage: PuppetPendingCertificateRequest (instance puppetserver1001:9100).
Dec 16 2025, 10:25 AM · Essential-Work, Data-Platform-SRE (2026.01.05 - 2026.01.23), sre-alert-triage

Dec 15 2025

LSobanski lowered the priority of T335431: Create an offline cookbook to take care of additional offline steps from Medium to Low.
Dec 15 2025, 3:50 PM · Infrastructure-Foundations, SRE-tools
LSobanski added a comment to T330997: Support locking cookbooks run except for switchover related cookbooks.

@Clement_Goubert is this still needed?

Dec 15 2025, 3:49 PM · ServiceOps new, SRE-tools, Infrastructure-Foundations, Datacenter-Switchover, SRE
LSobanski lowered the priority of T330843: reprepro uploads should trigger rsync apt job from Medium to Low.
Dec 15 2025, 3:47 PM · Packaging, Infrastructure-Foundations
LSobanski lowered the priority of T240843: Track services without a native systemd unit from Medium to Low.
Dec 15 2025, 3:42 PM · Infrastructure-Foundations, User-MoritzMuehlenhoff
LSobanski removed a project from T411503: x-provenance header: identify WMCS: Infrastructure-Foundations.
Dec 15 2025, 3:39 PM · Patch-For-Review, Traffic
LSobanski added a project to T412078: Alert in need of triage: SmartNotHealthy (instance sretest2006:9100): ops-codfw.
Dec 15 2025, 3:38 PM · SRE, ops-codfw, DC-Ops, Infrastructure-Foundations, sre-alert-triage
LSobanski renamed T412614: SystemdUnitFailed - gitlab2002 - backup-restore from SystemdUnitFailed to SystemdUnitFailed - gitlab2002 - backup-restore.
Dec 15 2025, 7:27 AM · collaboration-services

Dec 11 2025

LSobanski triaged T411895: gerrit behind CDN as High priority.
Dec 11 2025, 9:28 AM · Patch-For-Review, Gerrit, collaboration-services
LSobanski triaged T412016: Mailman failover process as Medium priority.
Dec 11 2025, 9:28 AM · collaboration-services

Dec 9 2025

LSobanski added a project to T409137: lists.wikimedia.org subscription email rejected by DKIM: collaboration-services.
Dec 9 2025, 7:41 AM · collaboration-services, Wikimedia-Mailing-lists, SRE, Infrastructure-Foundations
LSobanski created T412078: Alert in need of triage: SmartNotHealthy (instance sretest2006:9100).
Dec 9 2025, 7:40 AM · SRE, ops-codfw, DC-Ops, Infrastructure-Foundations, sre-alert-triage

Dec 8 2025

LSobanski removed a project from T386694: Replace k8s-controller-sidecars with built in Sidecar containers on k8s 1.31: collaboration-services.
Dec 8 2025, 4:50 PM · ServiceOps-good-first-task, ServiceOps new, Kubernetes, Prod-Kubernetes
LSobanski removed a project from T383553: Set cert-manager leader election namespace to cert-manager: collaboration-services.
Dec 8 2025, 4:50 PM · Machine-Learning-Team, Infrastructure-Foundations, ServiceOps new, Data-Platform-SRE, Kubernetes, Prod-Kubernetes
LSobanski removed a project from T387760: Migrate release template inheritance in helmfiles from YAML anchors to the inherit field: collaboration-services.
Dec 8 2025, 4:49 PM · Data-Platform-SRE, Kubernetes, Prod-Kubernetes, serviceops
LSobanski removed a project from T388387: Update kube-state-metrics for k8s 1.31: collaboration-services.
Dec 8 2025, 4:49 PM · Kubernetes, Prod-Kubernetes, serviceops
LSobanski removed a project from T388390: Ensure the correct helm version is used for each cluster: collaboration-services.
Dec 8 2025, 4:49 PM · ServiceOps-SharedInfra, ServiceOps new, Patch-For-Review, Data-Platform-SRE, Kubernetes, Prod-Kubernetes
LSobanski removed a project from T409137: lists.wikimedia.org subscription email rejected by DKIM: collaboration-services.
Dec 8 2025, 4:49 PM · collaboration-services, Wikimedia-Mailing-lists, SRE, Infrastructure-Foundations
LSobanski moved T403125: Investigate WMCS Magnum for GitLab runners from Incoming to Consultation on the collaboration-services board.
Dec 8 2025, 4:47 PM · Patch-For-Review, collaboration-services, Release-Engineering-Team (Priority Backlog 📥), GitLab (CI & Job Runners)
LSobanski moved T410572: Replace deprecated Phabricator Conduit API call by @ProdPasteBot with its stable equivalent from Incoming to Consultation on the collaboration-services board.
Dec 8 2025, 4:46 PM · collaboration-services, Phabricator
LSobanski assigned T411583: Gerrit backups are growing to ABran-WMF.
Dec 8 2025, 4:45 PM · collaboration-services, Gerrit
LSobanski moved T411904: ATS/Gerrit: validate TLS hosts for gerrit (revert workaround that skips validation) from Incoming to Backlog on the collaboration-services board.
Dec 8 2025, 4:44 PM · Traffic, Gerrit, collaboration-services
LSobanski triaged T411904: ATS/Gerrit: validate TLS hosts for gerrit (revert workaround that skips validation) as Medium priority.
Dec 8 2025, 4:44 PM · Traffic, Gerrit, collaboration-services
LSobanski changed the status of T411774: Requesting a new group allowing shell access to kafka-jumbo servers - with membership for JavierMonton from Open to Stalled.
Dec 8 2025, 3:57 PM · Data-Platform-SRE (2026.01.05 - 2026.01.23), Essential-Work, Infrastructure-Foundations
LSobanski triaged T411783: Move cloudweb hosts to cloud racks? as Low priority.
Dec 8 2025, 3:55 PM · Infrastructure-Foundations, netops, Striker, Horizon, cloud-services-team
LSobanski closed T409076: Public cloud account request for moving meta monitoring off of wikitech-static as Invalid.

Resolving based on the recent email communication.

Dec 8 2025, 10:07 AM · Infrastructure-Foundations
LSobanski changed the status of T341468: Migrate SRE repositories to GitLab from Open to Stalled.
Dec 8 2025, 10:06 AM · GitLab (Project Migration), collaboration-services
LSobanski changed the status of T349626: Migrate SRE repositories to GitLab - operations/alerts, a subtask of T341468: Migrate SRE repositories to GitLab, from Open to Stalled.
Dec 8 2025, 10:05 AM · GitLab (Project Migration), collaboration-services
LSobanski changed the status of T349626: Migrate SRE repositories to GitLab - operations/alerts from Open to Stalled.
Dec 8 2025, 10:05 AM · Observability-Alerting, GitLab (Project Migration), collaboration-services
LSobanski changed the status of T343707: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories from Open to Stalled.
Dec 8 2025, 10:05 AM · Projects-Cleanup, Release-Engineering-Team (Priority Backlog 📥), collaboration-services
LSobanski changed the status of T343707: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories, a subtask of T341468: Migrate SRE repositories to GitLab, from Open to Stalled.
Dec 8 2025, 10:05 AM · GitLab (Project Migration), collaboration-services
LSobanski placed T354479: ticket.wikimedia.org should page when down up for grabs.
Dec 8 2025, 10:05 AM · SRE-OnFire, collaboration-services, Znuny
LSobanski changed the status of T382161: Deploy a self-hosted public Matrix server instance, a subtask of T382159: Test Matrix, from Open to Stalled.
Dec 8 2025, 10:03 AM · Matrix, ERC
LSobanski changed the status of T382161: Deploy a self-hosted public Matrix server instance from Open to Stalled.

Removing myself as an assignee and stalling as I think this reflects the current state of this task.

Dec 8 2025, 10:03 AM · Matrix, ERC

Nov 24 2025

LSobanski updated the task description for T410634: Bring lists2001 into service.
Nov 24 2025, 6:45 PM · collaboration-services
LSobanski closed T354256: Mediawiki release archives downloads saturate the CPU and NIC on release hosts as Declined.

This has not happened since AFAIK, declining.

Nov 24 2025, 4:47 PM · collaboration-services
LSobanski added a comment to T256396: Create a runbook for switching CI master.

@hashar do you think there's still value in doing this with the Zuul upgrade coming soon?

Nov 24 2025, 4:43 PM · collaboration-services, Release-Engineering-Team (Priority Backlog 📥), Datacenter-Switchover, Continuous-Integration-Infrastructure
LSobanski removed a project from T371620: (some) Gitlab builds hanging: collaboration-services.
Nov 24 2025, 4:40 PM · Essential-Work, Release-Engineering-Team (Doing 😎), GitLab (CI & Job Runners)
LSobanski removed a project from T387886: Jobs on Digital Ocean Cloud Runners are being OOM killed: collaboration-services.
Nov 24 2025, 4:38 PM · Release-Engineering-Team (Priority Backlog 📥), User-brennen, GitLab (CI & Job Runners)
LSobanski removed a project from T383192: codesearch-write-config cronjob failing since 15 Dec: "RuntimeError: Unsure how to handle URL: https://codeberg.org/chdorner/CheckRegistrationEmailDomains": collaboration-services.
Nov 24 2025, 4:37 PM · VPS-project-Codesearch
LSobanski removed a project from T379110: Digital Ocean-based Cloud Gitlab Runners fail building Varnish: collaboration-services.
Nov 24 2025, 4:36 PM · Release-Engineering-Team
LSobanski removed a project from T163667: Fix UIDs for deployment server users: collaboration-services.
Nov 24 2025, 4:35 PM · serviceops-radar, Infrastructure-Foundations, Puppet, SRE
LSobanski moved T410418: Out of date directories / audit uploads and uploaders on releases.wikimedia.org from Incoming to Work in Progress (Tracking tasks) on the collaboration-services board.
Nov 24 2025, 4:31 PM · Release-Engineering-Team, collaboration-services
LSobanski assigned T410418: Out of date directories / audit uploads and uploaders on releases.wikimedia.org to Dzahn.
Nov 24 2025, 4:31 PM · Release-Engineering-Team, collaboration-services
LSobanski moved T289858: Use encrypted rsync for releases from Incoming to Backlog on the collaboration-services board.
Nov 24 2025, 4:28 PM · collaboration-services, MW-on-K8s, serviceops, SRE
LSobanski lowered the priority of T289858: Use encrypted rsync for releases from Medium to Low.
Nov 24 2025, 4:28 PM · collaboration-services, MW-on-K8s, serviceops, SRE
LSobanski moved T410510: Create an apt-staging VM in eqiad from Incoming to Backlog on the collaboration-services board.
Nov 24 2025, 4:26 PM · collaboration-services
LSobanski triaged T410510: Create an apt-staging VM in eqiad as Low priority.
Nov 24 2025, 4:26 PM · collaboration-services
LSobanski moved T410634: Bring lists2001 into service from Incoming to Work in Progress on the collaboration-services board.
Nov 24 2025, 4:25 PM · collaboration-services
LSobanski triaged T410634: Bring lists2001 into service as Medium priority.
Nov 24 2025, 4:25 PM · collaboration-services
LSobanski triaged T410384: Add an option to the reimage cookbook to also update firmware as Medium priority.
Nov 24 2025, 3:52 PM · SRE-tools, Infrastructure-Foundations, SRE
LSobanski moved T407844: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 from K8s to Backlog on the collaboration-services board.
Nov 24 2025, 3:18 PM · Upstream, Release-Engineering-Team, Gerrit, collaboration-services
LSobanski closed T341474: Migrate SRE repositories to GitLab - operations/cookbooks, a subtask of T341468: Migrate SRE repositories to GitLab, as Declined.
Nov 24 2025, 12:44 PM · GitLab (Project Migration), collaboration-services
LSobanski closed T341474: Migrate SRE repositories to GitLab - operations/cookbooks as Declined.

Not planning to do this anytime soon.

Nov 24 2025, 12:44 PM · GitLab (Project Migration), collaboration-services
LSobanski closed T343431: Migrate SRE repositories to GitLab - operations/dns as Declined.

Not planning to do this anytime soon.

Nov 24 2025, 12:43 PM · collaboration-services
LSobanski triaged T387831: Standardize failover procedures for Collab services as Medium priority.
Nov 24 2025, 12:37 PM · collaboration-services
LSobanski closed T297428: Audit actions for undo permissions / read restrictions bypass bug as Declined.

Declining as discussed above.

Nov 24 2025, 12:21 PM · MediaWiki-General, SecTeam-Processed, Security, Security-Team
LSobanski added a comment to T410858: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100).

Also eqiad-staging and codfw-staging.

Nov 24 2025, 9:34 AM · serviceops, sre-alert-triage
LSobanski created T410858: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100).
Nov 24 2025, 9:34 AM · serviceops, sre-alert-triage
LSobanski removed a project from T410823: ProbeDown - wdqs1015:443: collaboration-services.
Nov 24 2025, 8:28 AM · Data-Platform-SRE (2025.11.07 - 2025.11.28)

Nov 20 2025

LSobanski moved T391578: Releases failover process from Backlog to Work in Progress (Tracking tasks) on the collaboration-services board.
Nov 20 2025, 2:16 PM · collaboration-services
LSobanski created T410634: Bring lists2001 into service.
Nov 20 2025, 2:14 PM · collaboration-services
LSobanski created T410619: Alert in need of triage: SystemdUnitCrashLoop (instance grafana2001:9100).
Nov 20 2025, 12:39 PM · SRE Observability (FY2025/2026-Q2), Observability-Logging, sre-alert-triage