@hashar do you have specific plans on when you would like to upgrade to 3.9 and would you have a preferred order of executing the changes mentioned above?

Tue, May 7, 3:17 PM · Release-Engineering-Team, Gerrit, collaboration-services

LSobanski added a project to T284145: Clean up OTRS/Znuny addresses handles by gsuite: collaboration-services.

Tue, May 7, 10:58 AM · collaboration-services, Infrastructure-Foundations, Mail, Znuny, User-jbond

Mon, May 6

LSobanski closed T364307: ProbeDown - vrts2001 as Resolved.

This was @Arnoldokoth trying to upgrade the VRTS replica host.

Mon, May 6, 12:38 PM · collaboration-services

LSobanski renamed T364307: ProbeDown - vrts2001 from ProbeDown to ProbeDown - vrts2001.

Mon, May 6, 12:37 PM · collaboration-services

Wed, May 1

LSobanski created T363895: BGP status (instance cr2-eqord) - April 2024 - Equinix peering AS15830.

Wed, May 1, 7:43 AM · netops, Infrastructure-Foundations

LSobanski triaged T363760: ProbeDown (moscovium, rt) as Low priority.

Wed, May 1, 7:42 AM · collaboration-services

LSobanski moved T363760: ProbeDown (moscovium, rt) from Incoming to Work in Progress on the collaboration-services board.

Wed, May 1, 7:42 AM · collaboration-services

Tue, Apr 30

LSobanski renamed T362432: Upgrade etherpad.wikimedia.org to v2.0.3 from Upgrade etherpad.wikimedia.org to v2.0.2 to Upgrade etherpad.wikimedia.org to v2.0.3.

Tue, Apr 30, 10:01 AM · collaboration-services

Mon, Apr 29

LSobanski lowered the priority of T363009: Retire legalpad phabricator/phorge application? from Medium to Low.

Mon, Apr 29, 3:32 PM · Znuny, Trust-and-Safety, collaboration-services, Phabricator

LSobanski moved T363009: Retire legalpad phabricator/phorge application? from Incoming to Work in Progress on the collaboration-services board.

Mon, Apr 29, 3:32 PM · Znuny, Trust-and-Safety, collaboration-services, Phabricator

LSobanski assigned T363009: Retire legalpad phabricator/phorge application? to Dzahn.

Mon, Apr 29, 3:32 PM · Znuny, Trust-and-Safety, collaboration-services, Phabricator

LSobanski changed the status of T361170: Establish codex.wikimedia.org subdomain from Open to Stalled.

Mon, Apr 29, 3:31 PM · collaboration-services, Design-System-Team, Documentation, Codex

LSobanski moved T361170: Establish codex.wikimedia.org subdomain from Incoming to Backlog on the collaboration-services board.

Mon, Apr 29, 3:30 PM · collaboration-services, Design-System-Team, Documentation, Codex

LSobanski triaged T361170: Establish codex.wikimedia.org subdomain as Low priority.

Mon, Apr 29, 3:30 PM · collaboration-services, Design-System-Team, Documentation, Codex

LSobanski triaged T362933: Investigate OOM on vrts as Medium priority.

Mon, Apr 29, 3:29 PM · vrts, collaboration-services

LSobanski moved T362933: Investigate OOM on vrts from Incoming to Backlog on the collaboration-services board.

Mon, Apr 29, 3:29 PM · vrts, collaboration-services

LSobanski closed T363029: SystemdUnitFailed - gitlab - backup-restore as Resolved.

Mon, Apr 29, 3:28 PM · collaboration-services

LSobanski added a comment to T363196: Create test Gerrit instance.

@BCornwall what specific features do you need as it may influence how quickly this can happen?

Mon, Apr 29, 3:28 PM · collaboration-services, Release-Engineering-Team

LSobanski moved T363196: Create test Gerrit instance from Incoming to Backlog on the collaboration-services board.

Mon, Apr 29, 3:26 PM · collaboration-services, Release-Engineering-Team

LSobanski triaged T363196: Create test Gerrit instance as Low priority.

Mon, Apr 29, 3:25 PM · collaboration-services, Release-Engineering-Team

LSobanski moved T363564: make sure gitlab downtime works during version upgrades from Incoming to Backlog on the collaboration-services board.

Mon, Apr 29, 3:23 PM · collaboration-services

LSobanski triaged T363564: make sure gitlab downtime works during version upgrades as Medium priority.

Mon, Apr 29, 3:23 PM · collaboration-services

Fri, Apr 26

LSobanski triaged T363572: Reimage physical lists hosts to have public IPs as High priority.

Fri, Apr 26, 4:44 PM · collaboration-services, Wikimedia-Mailing-lists, SRE

LSobanski moved T363572: Reimage physical lists hosts to have public IPs from Incoming to Work in Progress on the collaboration-services board.

Fri, Apr 26, 4:43 PM · collaboration-services, Wikimedia-Mailing-lists, SRE

Thu, Apr 25

LSobanski added a comment to T331706: Migrate Mailman/lists to Bullseye/Bookworm.

Updating the host ownership in the Puppet role should also be part of this task.

Thu, Apr 25, 4:00 PM · Patch-For-Review, collaboration-services, Wikimedia-Mailing-lists, SRE

LSobanski added a comment to T284145: Clean up OTRS/Znuny addresses handles by gsuite.

Let's review if the new Znuny version enabled removal of unused emails and remove them if possible
If not, then let's filter the emails in the export script (in profile::mail::vrts)

Thu, Apr 25, 7:57 AM · collaboration-services, Infrastructure-Foundations, Mail, Znuny, User-jbond

Tue, Apr 23

LSobanski lowered the priority of T355976: VRTS Performance Improvements tracking from High to Medium.

Tue, Apr 23, 10:22 AM · Znuny, collaboration-services

LSobanski added a subtask for T355976: VRTS Performance Improvements tracking: T362933: Investigate OOM on vrts.

Tue, Apr 23, 10:21 AM · Znuny, collaboration-services

LSobanski added a parent task for T362933: Investigate OOM on vrts: T355976: VRTS Performance Improvements tracking.

Tue, Apr 23, 10:21 AM · vrts, collaboration-services

LSobanski moved T351202: stewards1001 / stewards2001: automatically subscribe stewards to mailman lists (was: Enable API access for Mailman3) from Work in Progress (Tracking tasks) to Work in Progress on the collaboration-services board.

Tue, Apr 23, 10:20 AM · Patch-For-Review, collaboration-services, SRE, Wikimedia-Mailing-lists, Stewards-Onboarding-Tool

LSobanski closed T362903: SystemdUnitFailed - vrts1001 - exim4 - clamav as Resolved.

Resolving as we have a follow up task.

Tue, Apr 23, 6:02 AM · collaboration-services

LSobanski added a comment to T363029: SystemdUnitFailed - gitlab - backup-restore.

This fired after T362989: Requests to prometheus pushgateway are timing out was resolved,

Tue, Apr 23, 5:58 AM · collaboration-services

LSobanski closed T362979: SystemdUnitFailed - gitlab - backup-restore as Resolved.

T362989: Requests to prometheus pushgateway are timing out was resolved.

Tue, Apr 23, 5:58 AM · collaboration-services

LSobanski renamed T363029: SystemdUnitFailed - gitlab - backup-restore from SystemdUnitFailed to SystemdUnitFailed - gitlab - backup-restore.

Tue, Apr 23, 5:53 AM · collaboration-services

LSobanski renamed T363113: ProbeDown - phab1004 from ProbeDown to ProbeDown - phab1004.

Tue, Apr 23, 5:52 AM · collaboration-services

Fri, Apr 19

LSobanski moved T334517: upgrade contint servers to bullseye from Work in Progress (Tracking tasks) to Work in Progress on the collaboration-services board.

Fri, Apr 19, 3:13 PM · Patch-For-Review, Release-Engineering-Team (Radar), collaboration-services

Thu, Apr 18

LSobanski added a comment to T357389: Alert in need of triage: BGP status (instance cr1-drmrs).

Additional BGP WARNING alert that showed up today:

Thu, Apr 18, 7:08 AM · Infrastructure-Foundations, netops, sre-alert-triage

Tue, Apr 16

LSobanski created T362611: Alert in need of triage: SystemdUnitFailed (instance db2200:9100).

Tue, Apr 16, 7:22 AM · DBA, sre-alert-triage

LSobanski added a project to T225694: Create cookbook to do `nodetool repair` across cassandra cluster: Cassandra.

@Eevans Tagging with Cassandra in case this may be of interest.

Tue, Apr 16, 6:51 AM · Cassandra, SRE-tools, User-Joe, SRE

LSobanski closed T362258: GitLab archival by Software Heritage as Resolved.

Tyler responded via e-mail so I think we're OK to close this one.

Tue, Apr 16, 6:43 AM · Release-Engineering-Team (Radar), collaboration-services, GitLab

Mon, Apr 15

LSobanski moved T362432: Upgrade etherpad.wikimedia.org to v2.0.3 from Incoming to Backlog on the collaboration-services board.

Mon, Apr 15, 3:35 PM · collaboration-services

LSobanski raised the priority of T355574: Many (all?) of the phabricator/tools scripts are in Python 2 from Lowest to Low.

Mon, Apr 15, 3:34 PM · collaboration-services, Python3-Porting, Phabricator, User-brennen, Release-Engineering-Team (Priority Backlog 📥)

LSobanski moved T355574: Many (all?) of the phabricator/tools scripts are in Python 2 from Incoming to Backlog on the collaboration-services board.

Mon, Apr 15, 3:34 PM · collaboration-services, Python3-Porting, Phabricator, User-brennen, Release-Engineering-Team (Priority Backlog 📥)

LSobanski lowered the priority of T355574: Many (all?) of the phabricator/tools scripts are in Python 2 from Medium to Lowest.

Mon, Apr 15, 3:34 PM · collaboration-services, Python3-Porting, Phabricator, User-brennen, Release-Engineering-Team (Priority Backlog 📥)

LSobanski moved T362258: GitLab archival by Software Heritage from Incoming to Consultation on the collaboration-services board.

Mon, Apr 15, 3:33 PM · Release-Engineering-Team (Radar), collaboration-services, GitLab

LSobanski added a comment to T362258: GitLab archival by Software Heritage.

No concerns from collaboration-services

Mon, Apr 15, 3:33 PM · Release-Engineering-Team (Radar), collaboration-services, GitLab

LSobanski moved T361277: gitlab: enforce 2fa for admins from Incoming to Work in Progress on the collaboration-services board.

Mon, Apr 15, 3:32 PM · Patch-For-Review, GitLab (Administration, Settings & Policy), User-brennen, Release-Engineering-Team, collaboration-services

LSobanski triaged T361277: gitlab: enforce 2fa for admins as High priority.

Mon, Apr 15, 3:32 PM · Patch-For-Review, GitLab (Administration, Settings & Policy), User-brennen, Release-Engineering-Team, collaboration-services

LSobanski assigned T361277: gitlab: enforce 2fa for admins to eoghan.

Mon, Apr 15, 3:32 PM · Patch-For-Review, GitLab (Administration, Settings & Policy), User-brennen, Release-Engineering-Team, collaboration-services

Fri, Apr 12

LSobanski closed T362334: SystemdUnitFailed - gitlab - backup-restore as Resolved.

Fri, Apr 12, 5:05 PM · collaboration-services

LSobanski added a comment to T362334: SystemdUnitFailed - gitlab - backup-restore.

Fallout from T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024, the alert has been silenced for additional 48h.

Fri, Apr 12, 5:05 PM · collaboration-services

LSobanski closed T358567: Switchover gitlab (gitlab2002 -> gitlab1003) - March 2024 as Declined.

We decided not to run the switchover at this time as the process has been sufficiently tested with the replica switchovers in T358559: Switchover gitlab replica (gitlab1004 -> gitlab1003) - March 2024 and the long maintenance window outweighs the benefit of switching over the production host.

Fri, Apr 12, 5:03 PM · collaboration-services

LSobanski closed T362322: ProbeDown - moscovium as Resolved.

This has recovered and the service is on a deprecation path so we won't be troubleshooting here.

Fri, Apr 12, 5:00 PM · collaboration-services

LSobanski triaged T362432: Upgrade etherpad.wikimedia.org to v2.0.3 as Low priority.

Fri, Apr 12, 4:59 PM · collaboration-services

LSobanski added a project to T362432: Upgrade etherpad.wikimedia.org to v2.0.3: collaboration-services.

Fri, Apr 12, 4:59 PM · collaboration-services

LSobanski created T362432: Upgrade etherpad.wikimedia.org to v2.0.3.

Fri, Apr 12, 4:58 PM · collaboration-services

LSobanski closed T354478: ticket.wikimedia.org down: upstream connect error or disconnect/reset before headers as Resolved.

I think we've monitored long enough. VRTS has been stable since the recent version upgrade.