Page MenuHomePhabricator

ayounsi (Arzhel Younsi)
Network Engineer

Projects (10)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Apr 3 2017, 6:23 PM (251 w, 4 d)
Availability
Available
IRC Nick
xionox
LDAP User
Ayounsi
MediaWiki User
AYounsi (WMF) [ Global Accounts ]

Recent Activity

Yesterday

ayounsi reassigned T300277: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs from ayounsi to RobH.

Please note the above diagram has a mistake, showing both routers connecting to PP:15/16 when cr1:xe-0/1/1 actually connects to Tata's port 11/12.

Fri, Jan 28, 8:16 AM · SRE, Infrastructure-Foundations, netops, ops-drmrs, DC-Ops

Wed, Jan 26

ayounsi triaged T300152: Investigate Ganeti in routed mode as Low priority.
Wed, Jan 26, 3:24 PM · Infrastructure-Foundations

Fri, Jan 21

ayounsi updated the task description for T299759: Install OpenGear console server (SCS) in new Eqiad cage.
Fri, Jan 21, 1:03 PM · SRE, DC-Ops, ops-eqiad
ayounsi added a comment to T299624: Switchover m1 master (db1159 -> db1128).

+1

Fri, Jan 21, 10:03 AM · Patch-For-Review, DBA

Thu, Jan 20

ayounsi closed T251156: add traceroute measurements to RIPE Atlas prometheus data, a subtask of T167689: Add RIPE atlas data to Prometheus, as Resolved.
Thu, Jan 20, 1:04 PM · observability, SRE
ayounsi closed T251156: add traceroute measurements to RIPE Atlas prometheus data as Resolved.

This is done, opened T299640 for further improvements.

Thu, Jan 20, 1:04 PM · Patch-For-Review, Observability-Metrics, Infrastructure-Foundations, netops, SRE
ayounsi triaged T299640: RIPE Atlas exporter improvements as Low priority.
Thu, Jan 20, 1:03 PM · observability, good first task, Infrastructure-Foundations

Wed, Jan 19

ayounsi closed T295819: ulsfo: (2) mx80s to become temp cr[34]-drmrs as Declined.

Not needed anymore.

Wed, Jan 19, 11:20 AM · SRE, netops, ops-ulsfo, DC-Ops, Infrastructure-Foundations
ayounsi added a comment to T254013: all network devices must run OpenSSH >= 7.2p1 but != 7.4p1.

Juniper bumped their recommended version to at least Junos 20 on a lot of platforms.

Wed, Jan 19, 10:50 AM · Infrastructure-Foundations, netops, SRE
ayounsi closed T299482: Paramiko > 2.8.1 incompatibility with some Juniper devices as Resolved.

Workaround pushed.

Wed, Jan 19, 10:46 AM · SRE, netops, Infrastructure-Foundations
ayounsi committed rOSHPd1fbc5c8df8a: Release v2.3.0 (authored by ayounsi).
Release v2.3.0
Wed, Jan 19, 10:38 AM
ayounsi committed rOSHOb92514b4687b: Update changelog for v0.3.0 (authored by ayounsi).
Update changelog for v0.3.0
Wed, Jan 19, 10:27 AM
ayounsi committed rOSHOeb50633f9fce: Force paramiko to 2.8.1 (authored by ayounsi).
Force paramiko to 2.8.1
Wed, Jan 19, 9:37 AM
ayounsi triaged T299482: Paramiko > 2.8.1 incompatibility with some Juniper devices as High priority.
Wed, Jan 19, 8:40 AM · SRE, netops, Infrastructure-Foundations
ayounsi reopened T297735: elastic1043.eqiad.wmnet stuck in power off state as "Open".

FYI the host is still set to "active" in Netbox.
https://netbox.wikimedia.org/dcim/devices/1366/

Wed, Jan 19, 8:01 AM · Discovery-Search (Current work)

Tue, Jan 18

ayounsi committed rOSHP0f02386e2a78: Update requirements (authored by ayounsi).
Update requirements
Tue, Jan 18, 6:17 PM
ayounsi committed rOBGP66a3ee408620: Add grafana-worldmap-panel (authored by ayounsi).
Add grafana-worldmap-panel
Tue, Jan 18, 3:38 PM
ayounsi closed T251184: Add Grafana worldmap panel as Resolved.

Done.

Tue, Jan 18, 2:02 PM · Observability-Metrics
ayounsi updated the language for P18765 (An Untitled Masterwork) from autodetect to bash.
Tue, Jan 18, 11:39 AM
ayounsi created P18765 (An Untitled Masterwork).
Tue, Jan 18, 11:39 AM
ayounsi committed rOSNEeb64ca46eb7a: LibreNMS report only count devices with no IP (authored by ayounsi).
LibreNMS report only count devices with no IP
Tue, Jan 18, 11:15 AM
ayounsi committed rOSNE7be77d8340ba: LibreNMS report, only log_info devices with no IP (authored by ayounsi).
LibreNMS report, only log_info devices with no IP
Tue, Jan 18, 10:42 AM

Mon, Jan 17

ayounsi committed rOSHO63bc1871a0e8: Bump Capirca to 2.0.4 (authored by ayounsi).
Bump Capirca to 2.0.4
Mon, Jan 17, 7:17 PM

Thu, Jan 13

ayounsi committed rOSNE8e549b7f49a4: LibreNMS report improvments (authored by ayounsi).
LibreNMS report improvments
Thu, Jan 13, 12:17 PM
ayounsi committed rOSNE9cd5aef3925f: Various reports improvements (authored by ayounsi).
Various reports improvements
Thu, Jan 13, 12:17 PM
ayounsi added a comment to T298980: Rack msw2-eqiad in new cage.

@ayounsi corrected et-0/1/0 Rolled fiber. has link.

Nice! and LLDP shows msw2 as neighbor.

Thu, Jan 13, 9:07 AM · SRE, ops-eqiad, DC-Ops
ayounsi added a comment to T222931: Netbox Reports Ideas and Requests.

[...]

Thu, Jan 13, 8:58 AM · Infrastructure-Foundations, netbox, User-crusnov, SRE-tools

Wed, Jan 12

ayounsi added a comment to T298980: Rack msw2-eqiad in new cage.

Thanks John!

Wed, Jan 12, 8:25 AM · SRE, ops-eqiad, DC-Ops

Tue, Jan 11

ayounsi updated the task description for T296966: eqiad: Master Tracking Ticket for eqiad expansion cage.
Tue, Jan 11, 3:48 PM · SRE, ops-eqiad, DC-Ops
ayounsi triaged T298980: Rack msw2-eqiad in new cage as Medium priority.
Tue, Jan 11, 3:46 PM · SRE, ops-eqiad, DC-Ops

Mon, Jan 10

ayounsi added a comment to T283771: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution).

Relying on parsing a website is often asking for troubles. Maybe we can also ask our account rep. for their recommendation (different API, etc).

Mon, Jan 10, 12:13 PM · Infrastructure-Foundations, Patch-For-Review, SRE-tools, SRE, netops, DC-Ops
ayounsi reassigned T295668: Update PDUs name-server config from ayounsi to RobH.

We usually use the FQDN for logging and NTP endpoints, see https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech#Setting_up_the_Configuration

Mon, Jan 10, 10:58 AM · SRE, ops-ulsfo
ayounsi added a comment to T283771: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution).

a quick look on GitHub shows 2 approaches:

Mon, Jan 10, 10:14 AM · Infrastructure-Foundations, Patch-For-Review, SRE-tools, SRE, netops, DC-Ops
ayounsi triaged T298869: msw-a8-eqiad potentially down as High priority.
Mon, Jan 10, 9:42 AM · SRE, ops-eqiad

Tue, Jan 4

ayounsi committed rOSHO59f6a7a8a773: Capirca: disable shade check (authored by ayounsi).
Capirca: disable shade check
Tue, Jan 4, 9:54 AM

Mon, Jan 3

ayounsi added a parent task for T273865: Investigate Capirca: Unknown Object (Task).
Mon, Jan 3, 3:06 PM · Infrastructure-Foundations, Patch-For-Review, SRE, netbox, homer, SRE-tools, netops
ayounsi closed T296935: Deprecate interface-range external as Resolved.

Deployed!

Mon, Jan 3, 3:00 PM · SRE, Infrastructure-Foundations, netops
ayounsi added a comment to T296271: Rack msw2-eqiad in cab A8 for configuration.

Great thanks!
I updated Netbox to reflect reality (as required so automation can work), and pushed its initial config.
Could you connect the mgmt port (em0) to ge-0/0/0 (to itself).

Mon, Jan 3, 9:15 AM · SRE, ops-eqiad
ayounsi triaged T298459: cr3-eqsin:xe-0/1/1 interface errors as Medium priority.
Mon, Jan 3, 7:38 AM · SRE, ops-eqsin

Dec 23 2021

ayounsi reassigned T296271: Rack msw2-eqiad in cab A8 for configuration from ayounsi to Cmjohnson.

The interface msw1:et-0/1/0 is alerting about CRC errors.

Dec 23 2021, 9:28 AM · SRE, ops-eqiad

Dec 17 2021

ayounsi added a comment to T263277: Collect netflow data for internal traffic.

In theory there should not be any PII data, but it would be safer to sanitize is nonetheless.

Dec 17 2021, 1:24 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi changed the status of T273865: Investigate Capirca from Stalled to In Progress.

Finally merged!

Dec 17 2021, 9:57 AM · Infrastructure-Foundations, Patch-For-Review, SRE, netbox, homer, SRE-tools, netops

Dec 15 2021

ayounsi added a comment to T263277: Collect netflow data for internal traffic.

Cool, only ip_version and region are useful here.

Dec 15 2021, 1:27 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi added a comment to T263277: Collect netflow data for internal traffic.

Am I right in assuming that this data has the same schema as the original netflow?

Dec 15 2021, 1:02 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE

Dec 14 2021

ayounsi added a comment to T263277: Collect netflow data for internal traffic.

Tests are successful:

Dec 14 2021, 2:19 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi closed T297595: Upgrade netflow VMs to Bullseye as Resolved.

All done!

Dec 14 2021, 2:15 PM · Patch-For-Review, SRE, netops, Infrastructure-Foundations
ayounsi closed T297595: Upgrade netflow VMs to Bullseye , a subtask of T263277: Collect netflow data for internal traffic, as Resolved.
Dec 14 2021, 2:15 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE

Dec 13 2021

ayounsi added a comment to T297588: connect 2nd cloudcontrol200x-dev NIC to vlan 2105.

Could we trunk the new vlan instead of using a 2nd physical port?

Dec 13 2021, 4:08 PM · SRE, Infrastructure-Foundations, netops, cloud-services-team (Kanban)
ayounsi closed T297609: Increase in prefix announcements from AS15169 as Resolved.

Thanks, we're already at 250000 for those. We usually set a high limit from the get go for route servers.

Dec 13 2021, 3:51 PM · Infrastructure-Foundations, SRE, netops
ayounsi added a subtask for T263277: Collect netflow data for internal traffic: T297595: Upgrade netflow VMs to Bullseye .
Dec 13 2021, 12:24 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi added a parent task for T297595: Upgrade netflow VMs to Bullseye : T263277: Collect netflow data for internal traffic.
Dec 13 2021, 12:24 PM · Patch-For-Review, SRE, netops, Infrastructure-Foundations
ayounsi triaged T297595: Upgrade netflow VMs to Bullseye as Medium priority.
Dec 13 2021, 12:24 PM · Patch-For-Review, SRE, netops, Infrastructure-Foundations

Dec 7 2021

ayounsi added a comment to T263277: Collect netflow data for internal traffic.

@Ottomata indeed we do have restriction on the producer side (it's the same tool as netflow, and can't HTTP POST) see T248865#6011043.

Dec 7 2021, 3:35 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi added a comment to T263277: Collect netflow data for internal traffic.

Sounds good!

  1. we can use "internal_flows" (not _netflow as netflow is a protocol).
  2. can I start this anytime, or we need to create the kafka topic somewhere?
Dec 7 2021, 2:02 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi added a comment to T295672: Use next-hop-self for iBGP sessions.

Ok, the fix from T295672#7531535 sounds good to me then!

Dec 7 2021, 1:42 PM · Sustainability (Incident Followup), Patch-For-Review, SRE, Infrastructure-Foundations, netops
ayounsi closed T284593: Create an alert for output discards on network devices as Resolved.

This is now set to alert to NOC through alertmanager.

Dec 7 2021, 8:41 AM · Infrastructure-Foundations, SRE, netops
ayounsi closed T284593: Create an alert for output discards on network devices, a subtask of T291627: Packet Drops on Eqiad ASW -> CR uplinks, as Resolved.
Dec 7 2021, 8:41 AM · SRE, Infrastructure-Foundations, netops
ayounsi changed the status of T295672: Use next-hop-self for iBGP sessions from Open to In Progress.
Dec 7 2021, 8:04 AM · Sustainability (Incident Followup), Patch-For-Review, SRE, Infrastructure-Foundations, netops
ayounsi added a comment to T295672: Use next-hop-self for iBGP sessions.

As a general note we need to be careful with rolling out config fixes in reaction to unexpected issues.
Even if it's thoroughly tested and I agree with your thorough proposal, it increases the config's complexity by tiny increments, making future changes (small or big) more risky.
As you pointed out, looking at our BGP confederation holistically is long due! (partially with T167841, possibly after looking at OSPF with T200277 to have sound foundations).

Dec 7 2021, 8:04 AM · Sustainability (Incident Followup), Patch-For-Review, SRE, Infrastructure-Foundations, netops
ayounsi added a comment to T296452: Upgrade Netbox to 3.1.

3.1 is out of beta, updated the task description accordingly.

Dec 7 2021, 7:16 AM · Infrastructure-Foundations, netbox
ayounsi updated the task description for T296452: Upgrade Netbox to 3.1.
Dec 7 2021, 7:16 AM · Infrastructure-Foundations, netbox

Dec 6 2021

ayounsi closed T295767: Rebuild ping* hosts with 10G disks as Resolved.

Alright, closing this for now then :)

Dec 6 2021, 4:59 PM · Patch-For-Review, netops, Infrastructure-Foundations, SRE
ayounsi added a comment to T263277: Collect netflow data for internal traffic.

Did you mean _not_ a hard requirement?

Yep, my bad :)

Dec 6 2021, 3:55 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi changed the status of T273865: Investigate Capirca from In Progress to Stalled.

Waiting for Capirca upstream to merge PRs.

Dec 6 2021, 3:40 PM · Infrastructure-Foundations, Patch-For-Review, SRE, netbox, homer, SRE-tools, netops
ayounsi added a comment to T296271: Rack msw2-eqiad in cab A8 for configuration.

Latest Junos recommended is 20.4R3-S1.3
I downloaded it to apt1001:/srv/junos/jinstall-ex-4300-20.4R3-S1.3-signed.tgz
You can also find it on https://webdownload.juniper.net/swdl/dl/secure/site/1/record/140793.html?pf=EX4300 if you have a Juniper account (and if you don't we should create you one :) )
Thanks!

Dec 6 2021, 12:31 PM · SRE, ops-eqiad
ayounsi added a comment to T263277: Collect netflow data for internal traffic.

@JAllemandou This is great, thanks! Note that we can tune sampling to adapt.

Dec 6 2021, 10:46 AM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE

Dec 2 2021

ayounsi triaged T296935: Deprecate interface-range external as Medium priority.
Dec 2 2021, 1:33 PM · SRE, Infrastructure-Foundations, netops
ayounsi added a comment to T294377: Q2:(Need By: TBD) rack/setup/install restbase202[456].codfw.wmnet.

From https://netbox.wikimedia.org/extras/reports/network.Network/

ge-6/0/26 Interface doesn't match its switch member: 5 on asw-b5-codfw

There was interface ge-6/0/26 configured on FPC5, I deleted it (as there is a ge-6/0/26 on fpc6), it should be good to go now.
I agree the Homer error message could be more clear though!

Dec 2 2021, 12:46 PM · Platform Team Workboards (Platform Engineering Reliability), SRE, RESTBase, ops-codfw, DC-Ops

Dec 1 2021

ayounsi changed the status of T295819: ulsfo: (2) mx80s to become temp cr[34]-drmrs from Open to Stalled.

Thanks I had a quick look and they both are healthy, all 8 interfaces show up as well.

Dec 1 2021, 9:26 AM · SRE, netops, ops-ulsfo, Infrastructure-Foundations, DC-Ops
ayounsi raised the priority of T294891: ps1-22-ulsfo Cord, Master_Cord_A, Active Power alerting from Medium to High.

Many sensors are now over threshold, see the red in: https://librenms.wikimedia.org/device/173/

Dec 1 2021, 8:22 AM · SRE, ops-ulsfo

Nov 30 2021

ayounsi added a comment to T263277: Collect netflow data for internal traffic.

@BTullis thanks! Real-time, would be a nice plus, but a hard requirement (unlike netflow).

Nov 30 2021, 6:04 PM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE
ayounsi added a comment to T296271: Rack msw2-eqiad in cab A8 for configuration.

I am confused, msw1-eqiad in A8 is already an EX-4300 48T. Do we want to replace with the same switch?

Nov 30 2021, 11:59 AM · SRE, ops-eqiad

Nov 29 2021

ayounsi committed rOHPU1d265c3c536d: Enable DHCP relay on mr1-codfw (authored by ayounsi).
Enable DHCP relay on mr1-codfw
Nov 29 2021, 8:51 PM

Nov 26 2021

ayounsi added a comment to T263277: Collect netflow data for internal traffic.

I went the "set a different sampling pipeline for internal flows" way with the above POC for the reasons mentioned in T263277#6491140.

Nov 26 2021, 10:28 AM · Data-Engineering-Kanban, Patch-For-Review, Data-Engineering, Traffic-Icebox, Infrastructure-Foundations, netops, SRE

Nov 25 2021

ayounsi committed rOSNBb5aaadd02853: netbox - cas: allow users with active=False (authored by Volans).
netbox - cas: allow users with active=False
Nov 25 2021, 5:05 PM
ayounsi committed rOSNBc422d03dd068: netbox - cas: only import cas view if we have cas enabled (authored by jbond).
netbox - cas: only import cas view if we have cas enabled
Nov 25 2021, 5:05 PM
ayounsi committed rOSNB9baad071c8df: Fix group assignement in CAS-SSO support (authored by Volans).
Fix group assignement in CAS-SSO support
Nov 25 2021, 5:05 PM
ayounsi committed rOSNBe2ee92f9c02e: netbox: ignore cas_configueration.py (authored by jbond).
netbox: ignore cas_configueration.py
Nov 25 2021, 5:05 PM
ayounsi committed rOSNB807ecbfa8292: Add CAS authentication support (authored by crusnov).
Add CAS authentication support
Nov 25 2021, 5:05 PM
ayounsi committed rOSNBa1f3ed2324f4: Add a passthrough configuration system (authored by crusnov).
Add a passthrough configuration system
Nov 25 2021, 5:05 PM
ayounsi committed rOSNBd90fcff70a25: add .gitreview file (authored by jbond).
add .gitreview file
Nov 25 2021, 5:05 PM
ayounsi committed rOSNBbc9a23233e84: Switch swagger to non-public mode (authored by crusnov).
Switch swagger to non-public mode
Nov 25 2021, 5:05 PM
ayounsi claimed T295767: Rebuild ping* hosts with 10G disks.

All 3 VMs got rebuilt with larger disks, but with the default Debian Buster.

Nov 25 2021, 3:28 PM · Patch-For-Review, netops, Infrastructure-Foundations, SRE
ayounsi committed rOHPU7cf456a25440: Move ping offload to new ping VMs (authored by ayounsi).
Move ping offload to new ping VMs
Nov 25 2021, 2:43 PM
ayounsi triaged T296452: Upgrade Netbox to 3.1 as Medium priority.
Nov 25 2021, 8:55 AM · Infrastructure-Foundations, netbox

Nov 24 2021

ayounsi updated the task description for T296411: cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet.
Nov 24 2021, 5:07 PM · SRE, netops, Infrastructure-Foundations, cloud-services-team (Kanban)
ayounsi edited projects for T296369: Kubernetes1018's eth negotiated speed is 10MB/s, added: ops-eqiad; removed Infrastructure-Foundations, netops.

That looks like a faulty cable or interface, over to DCops for troubleshooting, let us know if you need Netops help.

Nov 24 2021, 10:16 AM · ops-eqiad, SRE, serviceops

Nov 22 2021

ayounsi closed T295118: Can't commit on asw-b-codfw as Resolved.

Codfw repooled, everything is back to normal.

Nov 22 2021, 6:35 PM · SRE-swift-storage, ops-codfw, SRE, netops, Infrastructure-Foundations
ayounsi added a comment to T295118: Can't commit on asw-b-codfw.

The above command doesn't commit on a pre-provisioned VC.

Nov 22 2021, 1:14 PM · SRE-swift-storage, ops-codfw, SRE, netops, Infrastructure-Foundations

Nov 21 2021

ayounsi created P17784 (An Untitled Masterwork).
Nov 21 2021, 7:36 AM

Nov 19 2021

ayounsi added a comment to T295118: Can't commit on asw-b-codfw.

Hopefully we won't need to, but if asw1-b2-codfw needs to be rebooted, here are the impacted servers:
ms-be2041
ms-be2046
ms-be2031
ms-be2032
ms-fe2006
moss-be2002 (not active)
@MatthewVernon

Nov 19 2021, 10:40 AM · SRE-swift-storage, ops-codfw, SRE, netops, Infrastructure-Foundations
ayounsi added a comment to T295118: Can't commit on asw-b-codfw.

Current status:

  • IPv6 is still broken on asw-b7-codfw (for traffic local and transiting through the switch)
  • inet6 is disabled on cr2-codfw:ae2 (to row B)
    • That means row B have uplink redundancy for v4 but not v6
  • lvs2007 and codfw will stay depooled until Monday, when more intrusive remediation will be performed
    • codfw can be repooled if needed (eg. eqiad issue)
  • JTAC ticket can't be opened until T294792 is done
Nov 19 2021, 10:13 AM · SRE-swift-storage, ops-codfw, SRE, netops, Infrastructure-Foundations
ayounsi raised the priority of T163996: Icinga check for ipv6 host reachability from Medium to High.

Raising the priority to bring attention to this task, feel free to re-triage accordingly.

Nov 19 2021, 8:48 AM · SRE Observability, SRE

Nov 17 2021

ayounsi updated the task description for T295819: ulsfo: (2) mx80s to become temp cr[34]-drmrs.
Nov 17 2021, 7:30 PM · SRE, netops, ops-ulsfo, Infrastructure-Foundations, DC-Ops
ayounsi updated subscribers of T295118: Can't commit on asw-b-codfw.

For the record, there is also a link to lvs2007, after chatting with @BBlack on irc, the usual disable puppet then stop pybal is to do before the maintenance.

Nov 17 2021, 7:22 PM · SRE-swift-storage, ops-codfw, SRE, Infrastructure-Foundations, netops
ayounsi added a comment to T295819: ulsfo: (2) mx80s to become temp cr[34]-drmrs.

If you can take pictures of the front panels that could be useful to instruct remote hands when they get to drmrs too.

Nov 17 2021, 3:43 PM · SRE, netops, ops-ulsfo, DC-Ops, Infrastructure-Foundations
ayounsi updated subscribers of T295118: Can't commit on asw-b-codfw.
Nov 17 2021, 10:08 AM · SRE-swift-storage, ops-codfw, SRE, Infrastructure-Foundations, netops
ayounsi updated subscribers of T295118: Can't commit on asw-b-codfw.

This will cause a hard downtime for 6 servers (rack B7), for up to 1h, but most likely less:

Nov 17 2021, 10:06 AM · SRE-swift-storage, ops-codfw, SRE, Infrastructure-Foundations, netops
ayounsi assigned T295819: ulsfo: (2) mx80s to become temp cr[34]-drmrs to RobH.

mgmt ports to the mgmt switch please :)
Once we have this and console, we can check and upgrade them.

Nov 17 2021, 8:51 AM · SRE, netops, ops-ulsfo, DC-Ops, Infrastructure-Foundations

Nov 16 2021

ayounsi added a comment to T295118: Can't commit on asw-b-codfw.

That works for me, thanks, can you send a calendar invite? Note that the link in your comment doesn't point to any specific device.

Nov 16 2021, 5:46 PM · SRE-swift-storage, ops-codfw, SRE, Infrastructure-Foundations, netops