Page MenuHomePhabricator

LSobanski (Lukasz Sobanski)
Woo$

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Aug 31 2020, 5:40 PM (185 w, 8 h)
Availability
Available
LDAP User
LSobanski
MediaWiki User
LSobanski (WMF) [ Global Accounts ]

Recent Activity

Tue, Mar 12

LSobanski raised the priority of T323073: Make https://git.wikimedia.org not redirect to Phabricator Diffusion from Low to Medium.
Tue, Mar 12, 3:15 PM · Patch-For-Review, Diffusion, Release-Engineering-Team, collaboration-services
LSobanski triaged T358610: Update wmf/stable to Phorge upstream's 2023.49 stable release as High priority.
Tue, Mar 12, 2:52 PM · collaboration-services, User-brennen, Release-Engineering-Team, Phabricator

Mon, Mar 11

LSobanski added a comment to T359826: SystemdUnitFailed - gitlab1004.

Related to T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024

Mon, Mar 11, 6:02 PM · collaboration-services
LSobanski renamed T359826: SystemdUnitFailed - gitlab1004 from SystemdUnitFailed to SystemdUnitFailed - gitlab1004.
Mon, Mar 11, 5:58 PM · collaboration-services

Thu, Mar 7

LSobanski renamed T359526: SystemdUnitFailed - gitlab2002 from SystemdUnitFailed -gitlab2002 to SystemdUnitFailed - gitlab2002.
Thu, Mar 7, 12:18 PM · collaboration-services
LSobanski added a comment to T359526: SystemdUnitFailed - gitlab2002.

Ongoing maintenance.

Thu, Mar 7, 12:17 PM · collaboration-services
LSobanski renamed T359526: SystemdUnitFailed - gitlab2002 from SystemdUnitFailed to SystemdUnitFailed -gitlab2002.
Thu, Mar 7, 12:17 PM · collaboration-services

Wed, Mar 6

LSobanski added a comment to T359293: Alert in need of triage: ProbeDown (instance centrallog1002:6514).

Same set of alerts is firing for centrallog2002.

Wed, Mar 6, 9:27 AM · SRE Observability, sre-alert-triage
LSobanski created T359293: Alert in need of triage: ProbeDown (instance centrallog1002:6514).
Wed, Mar 6, 9:26 AM · SRE Observability, sre-alert-triage

Mon, Mar 4

LSobanski moved T359046: Bring back vrts in devtools from Incoming to Backlog on the collaboration-services board.
Mon, Mar 4, 4:28 PM · collaboration-services
LSobanski triaged T359046: Bring back vrts in devtools as Low priority.
Mon, Mar 4, 4:28 PM · collaboration-services
LSobanski triaged T358567: Switchover gitlab (gitlab2002 -> gitlab1003) - March 2024 as Medium priority.
Mon, Mar 4, 4:27 PM · collaboration-services
LSobanski moved T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024 from Incoming to Backlog on the collaboration-services board.
Mon, Mar 4, 4:27 PM · Patch-For-Review, collaboration-services
LSobanski triaged T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024 as Medium priority.
Mon, Mar 4, 4:27 PM · Patch-For-Review, collaboration-services
LSobanski assigned T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024 to eoghan.
Mon, Mar 4, 4:27 PM · Patch-For-Review, collaboration-services
LSobanski updated the task description for T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024.
Mon, Mar 4, 4:26 PM · Patch-For-Review, collaboration-services
LSobanski moved T358610: Update wmf/stable to Phorge upstream's 2023.49 stable release from Incoming to Backlog on the collaboration-services board.
Mon, Mar 4, 4:25 PM · collaboration-services, User-brennen, Release-Engineering-Team, Phabricator
LSobanski moved T358612: Investigate replacing Archiva with Gitlab repositories from Incoming to Consultation on the collaboration-services board.
Mon, Mar 4, 4:18 PM · Java-Scala-Standardization, Security, collaboration-services, Release-Engineering-Team, Data-Platform-SRE

Fri, Mar 1

LSobanski closed T358832: ProbeDown - vrts1002 as Resolved.

DB decommissioning - expected impact.

Fri, Mar 1, 11:32 AM · collaboration-services
LSobanski renamed T358832: ProbeDown - vrts1002 from ProbeDown - vrst1002 to ProbeDown - vrts1002.
Fri, Mar 1, 11:31 AM · collaboration-services

Wed, Feb 28

LSobanski updated the task description for T358612: Investigate replacing Archiva with Gitlab repositories.
Wed, Feb 28, 9:09 AM · Java-Scala-Standardization, Security, collaboration-services, Release-Engineering-Team, Data-Platform-SRE

Tue, Feb 27

LSobanski added a comment to T358237: Ganeti VM for contint migration.

@ayounsi we're on a tight schedule here as we're trying to get contint off of Buster by EOL. The approach of using a Ganeti VM was discussed over a few sessions with Moritz and RelEng as the fastest path forward and I would prefer not to change it.

Tue, Feb 27, 4:39 PM · collaboration-services, SRE, Continuous-Integration-Infrastructure, vm-requests
LSobanski added a subtask for T357547: ☂️ Northward Datacentre Switchover (March 2024) : T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024.
Tue, Feb 27, 12:09 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
LSobanski added a parent task for T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024: T357547: ☂️ Northward Datacentre Switchover (March 2024) .
Tue, Feb 27, 12:09 PM · Patch-For-Review, collaboration-services
LSobanski moved T357547: ☂️ Northward Datacentre Switchover (March 2024) from Incoming to Consultation on the collaboration-services board.
Tue, Feb 27, 12:09 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
LSobanski created T358567: Switchover gitlab (gitlab2002 -> gitlab1003) - March 2024.
Tue, Feb 27, 10:54 AM · collaboration-services
LSobanski renamed T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024 from Switchover gitlab (<old host> -> <new host>) - <month> <year> to Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024.
Tue, Feb 27, 10:52 AM · Patch-For-Review, collaboration-services
LSobanski created T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024.
Tue, Feb 27, 9:15 AM · Patch-For-Review, collaboration-services
LSobanski edited projects for T358506: Fix the Alert hosts Puppet catalogue to be compatible with Puppet 7, added: SRE Observability; removed SRE Observability (FY2023/2024-Q3).
Tue, Feb 27, 8:53 AM · SRE Observability (FY2023/2024-Q3), SRE

Mon, Feb 26

LSobanski closed T357719: Consider separating Gitlab code management and deb building management as Declined.

Right now I'll say the Debian way is the way we'll do things. The problems so far were teething issues that should go away in the longer run.

Mon, Feb 26, 5:03 PM · Traffic, collaboration-services
LSobanski triaged T358237: Ganeti VM for contint migration as High priority.
Mon, Feb 26, 4:55 PM · collaboration-services, SRE, Continuous-Integration-Infrastructure, vm-requests
LSobanski moved T358237: Ganeti VM for contint migration from Incoming to Backlog on the collaboration-services board.
Mon, Feb 26, 4:54 PM · collaboration-services, SRE, Continuous-Integration-Infrastructure, vm-requests
LSobanski renamed T357989: ProbeDown - etherpad1004 from ProbeDown to ProbeDown - etherpad1004.
Mon, Feb 26, 4:42 PM · collaboration-services
LSobanski closed T357958: ProbeDown - etherpad1003 as Resolved.

Etherpad has been upgraded to a new version. This alert was not related to the upgrade but we expect the new version to reduce these types of issues.

Mon, Feb 26, 4:40 PM · collaboration-services
LSobanski assigned T358237: Ganeti VM for contint migration to Dzahn.
Mon, Feb 26, 4:38 PM · collaboration-services, SRE, Continuous-Integration-Infrastructure, vm-requests

Fri, Feb 23

LSobanski added a project to T358237: Ganeti VM for contint migration: collaboration-services.
Fri, Feb 23, 7:56 AM · collaboration-services, SRE, Continuous-Integration-Infrastructure, vm-requests
LSobanski renamed T358227: SystemdUnitFailed - etherpad1004 - wmf_auto_restart_prometheus-etherpad-exporter.service from SystemdUnitFailed to SystemdUnitFailed - etherpad1004 - wmf_auto_restart_prometheus-etherpad-exporter.service.
Fri, Feb 23, 7:55 AM · collaboration-services

Wed, Feb 21

LSobanski added a comment to T354255: Alert in need of triage: AlertLintProblem (instance localhost:9123).

There are now four other similar alerts that are over a month old:

Wed, Feb 21, 9:50 AM · SRE Observability (FY2023/2024-Q3), sre-alert-triage

Tue, Feb 20

LSobanski added projects to T357896: ACPI kernel failure on debian installer last step: cloud-services-team, Infrastructure-Foundations.
Tue, Feb 20, 9:42 AM · Infrastructure-Foundations, cloud-services-team, User-aborrero, SRE
LSobanski renamed T357958: ProbeDown - etherpad1003 from ProbeDown to ProbeDown - etherpad1003.
Tue, Feb 20, 9:40 AM · collaboration-services

Mon, Feb 19

LSobanski moved T357572: scap install fails on new Phabricator/Phorge host due to missing user from Incoming to Backlog on the collaboration-services board.
Mon, Feb 19, 4:22 PM · User-brennen, Release-Engineering-Team (Now this 🫠), collaboration-services, Scap
LSobanski triaged T357572: scap install fails on new Phabricator/Phorge host due to missing user as Medium priority.
Mon, Feb 19, 4:21 PM · User-brennen, Release-Engineering-Team (Now this 🫠), collaboration-services, Scap
LSobanski moved T357612: Create a special-purpose Trusted Runner with Dockerfile frontend from Incoming to Backlog on the collaboration-services board.
Mon, Feb 19, 4:20 PM · Patch-For-Review, GitLab (CI & Job Runners), collaboration-services
LSobanski triaged T357679: PuppetDisabled - vrts1002 as Low priority.
Mon, Feb 19, 8:19 AM · collaboration-services
LSobanski moved T357679: PuppetDisabled - vrts1002 from Incoming to Work in Progress on the collaboration-services board.
Mon, Feb 19, 8:18 AM · collaboration-services
LSobanski created T357880: Alert in need of triage: Wikitech-static MW version up to date (instance wikitech-static.wikimedia.org).
Mon, Feb 19, 7:59 AM · wikitech.wikimedia.org, cloud-services-team, sre-alert-triage

Feb 15 2024

LSobanski closed T346607: GitLab email confirmation mail ends up in spam folder as Resolved.

Looking at the most recent comments, this should be OK to resolve, please reopen if you think otherwise.

Feb 15 2024, 4:59 PM · Release-Engineering-Team (Radar), GitLab (Infrastructure), collaboration-services
LSobanski closed T346607: GitLab email confirmation mail ends up in spam folder, a subtask of T355776: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines, as Resolved.
Feb 15 2024, 4:58 PM · Release-Engineering-Team (Radar), collaboration-services
LSobanski renamed T357679: PuppetDisabled - vrts1002 from PuppetDisabled to PuppetDisabled - vrts1002.
Feb 15 2024, 4:51 PM · collaboration-services
LSobanski created T357614: Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005).
Feb 15 2024, 10:32 AM · Data-Platform-SRE (2024.03.04 - 2024.03.24), sre-alert-triage
LSobanski updated the task description for T357612: Create a special-purpose Trusted Runner with Dockerfile frontend.
Feb 15 2024, 10:10 AM · Patch-For-Review, GitLab (CI & Job Runners), collaboration-services
LSobanski moved T357576: Upgrade VRTS To 6.5.6 from Incoming to Backlog on the collaboration-services board.
Feb 15 2024, 9:30 AM · Znuny, collaboration-services
LSobanski triaged T357576: Upgrade VRTS To 6.5.6 as Medium priority.
Feb 15 2024, 9:30 AM · Znuny, collaboration-services

Feb 14 2024

LSobanski added a comment to T357496: Alert in need of triage: Updater process (instance wdqs1022).

There are also alerts for wdqs1023 and wdqs1024.

Feb 14 2024, 8:30 AM · Data-Platform-SRE (2024.03.04 - 2024.03.24), Patch-For-Review, sre-alert-triage
LSobanski created T357496: Alert in need of triage: Updater process (instance wdqs1022).
Feb 14 2024, 8:30 AM · Data-Platform-SRE (2024.03.04 - 2024.03.24), Patch-For-Review, sre-alert-triage

Feb 13 2024

LSobanski created T357389: Alert in need of triage: BGP status (instance cr1-drmrs).
Feb 13 2024, 10:15 AM · Infrastructure-Foundations, netops, sre-alert-triage

Feb 12 2024

LSobanski moved T355502: phabricator_task_dump.service Failed on phab1004 from Work in Progress to Consultation on the collaboration-services board.
Feb 12 2024, 4:39 PM · Phabricator (2024-02-13), User-brennen, Patch-For-Review, Dumps-Generation, collaboration-services

Feb 10 2024

LSobanski renamed T356978: SystemdUnitFailed (gitlab1003 - backup-restore.service) from SystemdUnitFailed to SystemdUnitFailed (gitlab1003 - backup-restore.service).
Feb 10 2024, 10:21 AM · collaboration-services

Feb 8 2024

LSobanski added a project to T356951: PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL: Traffic.
Feb 8 2024, 4:12 PM · Traffic, SRE

Feb 7 2024

LSobanski moved T355980: [vrts] Move attachment storage out of mysql from Backlog to Work in Progress on the collaboration-services board.
Feb 7 2024, 5:06 PM · Znuny, collaboration-services
LSobanski merged task T138915: OTRS database is "too large" into T355980: [vrts] Move attachment storage out of mysql.
Feb 7 2024, 1:09 PM · collaboration-services, vrts, database-backups, DBA, Znuny
LSobanski merged T138915: OTRS database is "too large" into T355980: [vrts] Move attachment storage out of mysql.
Feb 7 2024, 1:09 PM · Znuny, collaboration-services
LSobanski updated the task description for T355980: [vrts] Move attachment storage out of mysql.
Feb 7 2024, 1:09 PM · Znuny, collaboration-services
LSobanski added a comment to T316421: Upgrade etherpad.wikimedia.org to v1.9.7.

I updated the description to reflect the new Etherpad release (1.9.7). See below for a list of changes:

  • Notable enhancements and fixes
    • Added Live Plugin Manager: Plugins are now installed into a separate folder on the host system. This folder is called plugin_packages.
    • That way the plugins are separated from the normal etherpad installation.
    • Make repairPad.js more verbose
    • Fixed favicon not being loaded correctly
Feb 7 2024, 9:56 AM · User-notice-archive, collaboration-services, SRE, Wikimedia-Etherpad
LSobanski renamed T316421: Upgrade etherpad.wikimedia.org to v1.9.7 from Upgrade etherpad.wikimedia.org to v1.9.6 to Upgrade etherpad.wikimedia.org to v1.9.7.
Feb 7 2024, 9:55 AM · User-notice-archive, collaboration-services, SRE, Wikimedia-Etherpad
LSobanski added a comment to T356077: Inbound mail to phabricator doesn't work.

What's documented is creating tasks by email, which also fails but in a different way:

Feb 7 2024, 8:38 AM · Phabricator, collaboration-services

Feb 6 2024

LSobanski added a member for Incident Tooling: LSobanski.
Feb 6 2024, 4:15 PM
LSobanski moved T356530: clean up phabricator test host situation from Incoming to Backlog on the collaboration-services board.
Feb 6 2024, 2:30 PM · collaboration-services
LSobanski triaged T356530: clean up phabricator test host situation as Medium priority.
Feb 6 2024, 2:30 PM · collaboration-services
LSobanski removed a project from T337949: Add security.txt to Wikimedia sites? (2023 edition): SRE.

Removing SRE, please add us back when the decision to implement this has been made.

Feb 6 2024, 8:33 AM · Patch-For-Review, SecTeam-Processed, Documentation, WMF-General-or-Unknown, Security-Team, Security, Wikimedia-Apache-configuration
LSobanski moved T354479: ticket.wikimedia.org should page when down from Work in Progress to Backlog on the collaboration-services board.
Feb 6 2024, 8:23 AM · SRE-OnFire, collaboration-services, Znuny
LSobanski moved T348454: VRTS Post Upgrade Actions from Backlog to Work in Progress on the collaboration-services board.
Feb 6 2024, 8:23 AM · Vuln-VulnComponent, SecTeam-Processed, vrts, collaboration-services, Security

Feb 5 2024

LSobanski moved T356426: VRTS System Log Error from Incoming to Work in Progress on the collaboration-services board.
Feb 5 2024, 4:20 PM · vrts, collaboration-services
LSobanski triaged T356426: VRTS System Log Error as Medium priority.
Feb 5 2024, 4:20 PM · vrts, collaboration-services
LSobanski triaged T355574: Many (all?) of the phabricator/tools scripts are in Python 2 as Medium priority.
Feb 5 2024, 12:38 PM · Patch-For-Review, collaboration-services, Python3-Porting, Phabricator, User-brennen, Release-Engineering-Team (Priority Backlog 📥)

Feb 1 2024

LSobanski lowered the priority of T355776: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines from High to Medium.
Feb 1 2024, 5:54 PM · Release-Engineering-Team (Radar), collaboration-services
LSobanski added a project to T356376: Etherpad need restore to previous revision : collaboration-services.
Feb 1 2024, 9:18 AM · collaboration-services, SRE, Wikimedia-Etherpad
LSobanski closed T356245: ProbeDown - vrts1001 as Resolved.

OOM-killed. VM RAM was increased to 12GB in T355977: [vrts] Increase memory allocation of vrts instances.

Feb 1 2024, 9:04 AM · Znuny, collaboration-services
LSobanski moved T355977: [vrts] Increase memory allocation of vrts instances from Backlog to Work in Progress on the collaboration-services board.
Feb 1 2024, 9:03 AM · Znuny, collaboration-services

Jan 31 2024

LSobanski moved T356140: blackbox alerts: add instance names to tickets created by alerting from Incoming to Consultation on the collaboration-services board.
Jan 31 2024, 2:33 PM · Observability-Alerting, collaboration-services
LSobanski triaged T355776: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines as High priority.
Jan 31 2024, 2:32 PM · Release-Engineering-Team (Radar), collaboration-services
LSobanski triaged T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines as High priority.
Jan 31 2024, 2:32 PM · collaboration-services
LSobanski moved T355776: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines from Incoming to Work in Progress on the collaboration-services board.
Jan 31 2024, 2:32 PM · Release-Engineering-Team (Radar), collaboration-services
LSobanski renamed T356245: ProbeDown - vrts1001 from ProbeDown vrts1001:1443/vrts1001:25 to ProbeDown - vrts1001.
Jan 31 2024, 9:38 AM · Znuny, collaboration-services

Jan 30 2024

LSobanski added a comment to T337949: Add security.txt to Wikimedia sites? (2023 edition).

AS the original task was declined without comment, it would be helpful to understand what the input we're looking for in this task is (from SRE but also in general)? And who would be the decision maker, Security-Team?

Jan 30 2024, 8:36 AM · Patch-For-Review, SecTeam-Processed, Documentation, WMF-General-or-Unknown, Security-Team, Security, Wikimedia-Apache-configuration

Jan 29 2024

LSobanski moved T310822: Create a Prometheus exporter for Znuny metrics of interest to the VRT from Work in Progress to Backlog on the collaboration-services board.
Jan 29 2024, 4:50 PM · Patch-For-Review, Observability-Metrics, vrts, collaboration-services, Znuny
LSobanski moved T347004: Create a staging apt repository for CI-based builds of Debian packages from Work in Progress to Backlog on the collaboration-services board.
Jan 29 2024, 4:50 PM · collaboration-services, GitLab (CI & Job Runners), serviceops
LSobanski moved T355814: VRTS Database Growth Rate from Incoming to Backlog on the collaboration-services board.
Jan 29 2024, 4:50 PM · Data-Persistence, collaboration-services
LSobanski triaged T355814: VRTS Database Growth Rate as Medium priority.
Jan 29 2024, 4:49 PM · Data-Persistence, collaboration-services
LSobanski moved T355980: [vrts] Move attachment storage out of mysql from Incoming to Backlog on the collaboration-services board.
Jan 29 2024, 4:46 PM · Znuny, collaboration-services
LSobanski moved T355977: [vrts] Increase memory allocation of vrts instances from Incoming to Backlog on the collaboration-services board.
Jan 29 2024, 4:46 PM · Znuny, collaboration-services
LSobanski moved T355979: [vrts] Switch ticket index module from IndexAccelerator::RuntimeDB to StaticDB from Incoming to Backlog on the collaboration-services board.
Jan 29 2024, 4:46 PM · Patch-For-Review, Znuny, collaboration-services
LSobanski triaged T355977: [vrts] Increase memory allocation of vrts instances as Medium priority.
Jan 29 2024, 4:45 PM · Znuny, collaboration-services
LSobanski triaged T355980: [vrts] Move attachment storage out of mysql as Medium priority.
Jan 29 2024, 4:45 PM · Znuny, collaboration-services
LSobanski triaged T355979: [vrts] Switch ticket index module from IndexAccelerator::RuntimeDB to StaticDB as High priority.
Jan 29 2024, 4:45 PM · Patch-For-Review, Znuny, collaboration-services
LSobanski moved T355516: Refine alert manager silences for Gerrit upgrades from Incoming to Backlog on the collaboration-services board.
Jan 29 2024, 4:41 PM · collaboration-services, Release-Engineering-Team, Gerrit
LSobanski triaged T355516: Refine alert manager silences for Gerrit upgrades as Low priority.
Jan 29 2024, 4:41 PM · collaboration-services, Release-Engineering-Team, Gerrit
LSobanski moved T356077: Inbound mail to phabricator doesn't work from Incoming to Backlog on the collaboration-services board.
Jan 29 2024, 4:41 PM · Phabricator, collaboration-services
LSobanski triaged T356077: Inbound mail to phabricator doesn't work as Low priority.
Jan 29 2024, 4:41 PM · Phabricator, collaboration-services
LSobanski added a comment to T356077: Inbound mail to phabricator doesn't work.

This feature might have been non-operational even before the upgrade. It's not advertised and we don't know of any complaints. Worth looking into but not high priority.

Jan 29 2024, 4:40 PM · Phabricator, collaboration-services