Page MenuHomePhabricator

ayounsi (Arzhel Younsi)
Staff Network SRE

Projects (15)

Today

  • No visible events.

Tomorrow

  • No visible events.

Friday

  • No visible events.

User Details

User Since
Apr 3 2017, 6:23 PM (496 w, 1 d)
Availability
Available
IRC Nick
xionox
LDAP User
Ayounsi
MediaWiki User
AYounsi (WMF) [ Global Accounts ]

Recent Activity

Mon, Oct 5

ayounsi added a comment to T440071: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field.

Good idea, before we start entering data we need to add validators to prevent entry mistakes (regex for the format, maybe coherence in the numbering) and prevent ID-less racks in the relevant sites.

Mon, Oct 5, 9:23 AM · SRE, ops-codfw, DC-Ops
ayounsi added a comment to T440070: Enter Equinix's Rack Location in Netbox's "Facility ID" Field.

Good idea, before we start entering data we need to add validators to prevent entry mistakes (regex for the format, maybe coherence in the numbering) and prevent ID-less racks in the relevant sites.

Mon, Oct 5, 9:23 AM · SRE, ops-eqiad, DC-Ops
ayounsi updated subscribers of T439947: eqsin-codfw HE packet loss/latency - oct 2026.

@wiki_willy can you ask for credits on that one too ? HE ticket 7282921 for circuit dc4841.dal5.sin1 showing issues between 2026-10-02 06:20 and 2026-10-03 04:40 with a 4h improvement window in the middle.

Mon, Oct 5, 7:33 AM · Infrastructure-Foundations, netops

Fri, Oct 2

ayounsi added a comment to T439947: eqsin-codfw HE packet loss/latency - oct 2026.

Thank you for this report. There is currently maintenance being performed on some of our trans-Pacific circuits which may be impacting your traffic. It should be completed by 1300 UTC.

Fri, Oct 2, 9:03 AM · Infrastructure-Foundations, netops
ayounsi added a parent task for T439947: eqsin-codfw HE packet loss/latency - oct 2026: Unknown Object (Task).
Fri, Oct 2, 9:01 AM · Infrastructure-Foundations, netops
ayounsi updated subscribers of T439947: eqsin-codfw HE packet loss/latency - oct 2026.
Fri, Oct 2, 9:00 AM · Infrastructure-Foundations, netops
ayounsi added a project to T439947: eqsin-codfw HE packet loss/latency - oct 2026: netops.
Fri, Oct 2, 8:58 AM · Infrastructure-Foundations, netops
ayounsi updated the task description for T439947: eqsin-codfw HE packet loss/latency - oct 2026.
Fri, Oct 2, 8:58 AM · Infrastructure-Foundations, netops
ayounsi created T439947: eqsin-codfw HE packet loss/latency - oct 2026.
Fri, Oct 2, 8:46 AM · Infrastructure-Foundations, netops
ayounsi updated subscribers of T439644: Grant access to analytics-privatedata-users for Eniroomand-WMF.

Please open a new ticket, the person in charge of access requests changes every 7 days (it's now @herron)

Fri, Oct 2, 5:46 AM · Data-Engineering-Radar, Data-Engineering, SRE-Access-Requests, SRE

Thu, Oct 1

ayounsi reassigned T417873: eqiad: upgrade routers (2026) from cmooney to Papaul.

@Papaul, are you (or someone else in DCops) ok to take care of mr1-eqiad?

Thu, Oct 1, 2:31 PM · Infrastructure-Foundations, netops
ayounsi added a comment to T417873: eqiad: upgrade routers (2026).

The new JTAC recommended version for the SRX300 is now Junos 24.4R2-S3.

Thu, Oct 1, 2:29 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T417873: eqiad: upgrade routers (2026).
Thu, Oct 1, 2:28 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T417873: eqiad: upgrade routers (2026).
Thu, Oct 1, 2:27 PM · Infrastructure-Foundations, netops
ayounsi added a comment to T417871: codfw: upgrade routers (2026).

Let's use the upgrade opportunity to enable BGP RIB sharding on one of the two routers T320264: Junos: investigate BGP rib sharding

Thu, Oct 1, 2:27 PM · Infrastructure-Foundations, netops
ayounsi added a comment to T416563: eqsin: upgrade routers (2026).

Let's use the upgrade opportunity to enable BGP RIB sharding on cr2-eqsin T320264: Junos: investigate BGP rib sharding

Thu, Oct 1, 2:27 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T320264: Junos: investigate BGP rib sharding.
Thu, Oct 1, 2:26 PM · netops, Infrastructure-Foundations
ayounsi closed T439644: Grant access to analytics-privatedata-users for Eniroomand-WMF as Resolved.

Give it 30min for the change to propagate and you should be good. Please re-open if there is any issue.

Thu, Oct 1, 1:55 PM · Data-Engineering-Radar, Data-Engineering, SRE-Access-Requests, SRE
ayounsi closed T439665: Add FIDO-backed SSH keys for kharlan as Resolved.

All done, give it 30min for the removal to propagate. Please reopen if any issue.

Thu, Oct 1, 12:03 PM · SRE, SRE-Access-Requests
ayounsi assigned T439646: Bing Webmaster Tools access request for Ash Han to LSobanski.

@LSobanski is that something you can help with ?

Thu, Oct 1, 9:06 AM · SRE
ayounsi closed T439138: Requesting access to deployment for Tsevener as Resolved.

Give it 30min for the change to propagate. Please re-open the task if you have any issue.

Thu, Oct 1, 7:59 AM · SRE, SRE-Access-Requests
ayounsi closed T438926: Requesting access to analytics-privatedata-users and deployment for lwatson as Resolved.

Give it ~30min for the change to propagate and give it a try. Please re-open the task if any issue. Thanks

Thu, Oct 1, 7:46 AM · SRE, SRE-Access-Requests

Wed, Sep 30

ayounsi added a project to T439644: Grant access to analytics-privatedata-users for Eniroomand-WMF: Data-Engineering.

@eniroomand Please familiarize yourself with https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#User_responsibilities
Could you also get your manager's approval on this task ?

Wed, Sep 30, 3:35 PM · Data-Engineering-Radar, Data-Engineering, SRE-Access-Requests, SRE
ayounsi updated the task description for T438926: Requesting access to analytics-privatedata-users and deployment for lwatson.
Wed, Sep 30, 3:02 PM · SRE, SRE-Access-Requests
ayounsi updated the task description for T439138: Requesting access to deployment for Tsevener.
Wed, Sep 30, 3:01 PM · SRE, SRE-Access-Requests

Tue, Sep 29

ayounsi closed T439202: Junos: ae speed not reported correctly through gNMI as Resolved.

Deployed!

Tue, Sep 29, 9:26 AM · Infrastructure-Foundations, netops
ayounsi claimed T439202: Junos: ae speed not reported correctly through gNMI.

Great point, I sent https://gerrit.wikimedia.org/r/c/operations/alerts/+/1345934 for that.

Tue, Sep 29, 9:07 AM · Infrastructure-Foundations, netops
ayounsi added a parent task for T416444: 2026 Junos upgrade: T439136: Juniper hostkey-algorithm deprecation.
Tue, Sep 29, 8:56 AM · Traffic, Infrastructure-Foundations, netops
ayounsi added a subtask for T439136: Juniper hostkey-algorithm deprecation: T416444: 2026 Junos upgrade.
Tue, Sep 29, 8:56 AM · netops, Infrastructure-Foundations
ayounsi changed the status of T439136: Juniper hostkey-algorithm deprecation from Open to Stalled.

Marking it as stalled until we have upgraded all the devices.

Tue, Sep 29, 8:55 AM · netops, Infrastructure-Foundations
cmooney awarded T439134: Restart lsw1-a2-codfw for T437984 a Y So Serious token.
Tue, Sep 29, 8:09 AM · Infrastructure-Foundations, netops

Mon, Sep 28

ayounsi triaged T439202: Junos: ae speed not reported correctly through gNMI as Medium priority.
Mon, Sep 28, 2:45 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T439408: Netbox: setup netbox-bgp.
Mon, Sep 28, 12:17 PM · Patch-For-Review, Infrastructure-Foundations, netbox
ayounsi triaged T439408: Netbox: setup netbox-bgp as Medium priority.
Mon, Sep 28, 12:17 PM · Patch-For-Review, Infrastructure-Foundations, netbox
ayounsi created T439408: Netbox: setup netbox-bgp.
Mon, Sep 28, 12:16 PM · Patch-For-Review, Infrastructure-Foundations, netbox
ayounsi closed T437984: Junos file descriptors exhaustion as Resolved.

All done!

Mon, Sep 28, 8:00 AM · Infrastructure-Foundations, netops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Mon, Sep 28, 8:00 AM · Infrastructure-Foundations, netops
ayounsi closed T439134: Restart lsw1-a2-codfw for T437984, a subtask of T437984: Junos file descriptors exhaustion, as Resolved.
Mon, Sep 28, 7:58 AM · Infrastructure-Foundations, netops
ayounsi closed T439134: Restart lsw1-a2-codfw for T437984 as Resolved.

Well, CyrusOne took care of the reboot with the power outage.

Mon, Sep 28, 7:58 AM · Infrastructure-Foundations, netops
ayounsi closed T327300: Cookbook for rack depool as Resolved.

Going to close that one, the pool and depool actions have been tested, now supports also the "manual" policy. Some roles don't have policies, (for example backup or lvs hosts) but that's up to the service owners if they want to document it.

Mon, Sep 28, 7:49 AM · Patch-For-Review, Infrastructure-Foundations, SRE-tools
ayounsi updated the task description for T439134: Restart lsw1-a2-codfw for T437984.
Mon, Sep 28, 7:45 AM · Infrastructure-Foundations, netops

Fri, Sep 25

ayounsi updated the task description for T439143: Requesting access to Deployment for GreyOlson/GOlson-WMF.
Fri, Sep 25, 1:21 PM · Patch-For-Review, SRE, SRE-Access-Requests
ayounsi updated the task description for T439138: Requesting access to deployment for Tsevener.
Fri, Sep 25, 1:20 PM · SRE, SRE-Access-Requests
ayounsi updated the task description for T439138: Requesting access to deployment for Tsevener.
Fri, Sep 25, 1:15 PM · SRE, SRE-Access-Requests
ayounsi updated the task description for T439143: Requesting access to Deployment for GreyOlson/GOlson-WMF.
Fri, Sep 25, 1:09 PM · Patch-For-Review, SRE, SRE-Access-Requests
ayounsi added a comment to T439143: Requesting access to Deployment for GreyOlson/GOlson-WMF.

@ayounsi I just had one (ed25519) already for other access, figured it would be a little easier for me to have the two separate style ones!

Fri, Sep 25, 1:08 PM · Patch-For-Review, SRE, SRE-Access-Requests
ayounsi updated the task description for T439143: Requesting access to Deployment for GreyOlson/GOlson-WMF.
Fri, Sep 25, 12:51 PM · Patch-For-Review, SRE, SRE-Access-Requests
ayounsi updated subscribers of T439143: Requesting access to Deployment for GreyOlson/GOlson-WMF.

@GOlson-WMF could you use an ed25519 ssh key? Or detail why you can't use one.

Fri, Sep 25, 12:51 PM · Patch-For-Review, SRE, SRE-Access-Requests
ayounsi updated subscribers of T439138: Requesting access to deployment for Tsevener.

@thcipriani @dancy do you approve this request ?
@Tsevener can you sign https://phabricator.wikimedia.org/L3 ?

Fri, Sep 25, 12:48 PM · SRE, SRE-Access-Requests
ayounsi updated the task description for T439138: Requesting access to deployment for Tsevener.
Fri, Sep 25, 12:36 PM · SRE, SRE-Access-Requests
ayounsi added a comment to T388138: PDUs in Active status missing IP address information in NetBox.

Randomly found this task.

Fri, Sep 25, 12:17 PM · DC-Ops, Observability-Metrics
ayounsi reopened T438161: QFY2627 :rack/setup/install ml-serve1016 as "Open".

There is an outstanding change on the switch for that host:

Change for lsw1-e5-eqiad.mgmt.eqiad.wmnet:
Fri, Sep 25, 5:40 AM · Data-Platform-SRE, SRE, ops-eqiad, DC-Ops
ayounsi created T439202: Junos: ae speed not reported correctly through gNMI.
Fri, Sep 25, 5:25 AM · Infrastructure-Foundations, netops

Thu, Sep 24

ayounsi created T439166: Junos 25 and gNMI metrics.
Thu, Sep 24, 7:14 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T439136: Juniper hostkey-algorithm deprecation.
Thu, Sep 24, 2:05 PM · netops, Infrastructure-Foundations
ayounsi added a comment to T439136: Juniper hostkey-algorithm deprecation.
bast6003:~$ ssh -o HostKeyAlgorithms=ssh-dss -o PubkeyAcceptedAlgorithms=ssh-dss cr1-drmrs.wikimedia.org
Unable to negotiate with 2a02:ec80:600:ffff::1 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
Thu, Sep 24, 1:58 PM · netops, Infrastructure-Foundations
ayounsi created T439136: Juniper hostkey-algorithm deprecation.
Thu, Sep 24, 1:24 PM · netops, Infrastructure-Foundations
ayounsi added a parent task for T439134: Restart lsw1-a2-codfw for T437984: T437984: Junos file descriptors exhaustion.
Thu, Sep 24, 12:58 PM · Infrastructure-Foundations, netops
ayounsi added a subtask for T437984: Junos file descriptors exhaustion: T439134: Restart lsw1-a2-codfw for T437984.
Thu, Sep 24, 12:58 PM · Infrastructure-Foundations, netops
ayounsi created T439134: Restart lsw1-a2-codfw for T437984.
Thu, Sep 24, 12:58 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T371889: Upgrade Netbox to 4.7.x.
Thu, Sep 24, 12:45 PM · netbox, Infrastructure-Foundations
ayounsi created T439130: Netbox: reports alerting issues.
Thu, Sep 24, 12:25 PM · Infrastructure-Foundations, netbox
ayounsi added a comment to T438963: Possible Netbox incorrection on device type.

Looks like some of the checks are not alerting correctly: https://netbox.wikimedia.org/extras/scripts/results/398218/

Thu, Sep 24, 12:19 PM · SRE, ops-eqiad, DC-Ops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Thu, Sep 24, 9:32 AM · Infrastructure-Foundations, netops
ayounsi created T439099: Cookbook for AlertManager silence by key=value.
Thu, Sep 24, 7:36 AM · Observability-Alerting
ayounsi assigned T438269: tcpircbot shouldn't crash if it receives a message with a newline to tappof.

[clinic duty] boldly assigning the task to Tiziano as they sent a patch.

Thu, Sep 24, 5:47 AM · Observability-Alerting, Patch-For-Review

Tue, Sep 22

ayounsi updated subscribers of T395032: Cookbook sre.hosts.remove_downtime does not remove silences.

Working on an alerting feature for one of "my" cookbooks (https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1343990) I was wondering if o11y was planning on improving the remove-downtime cookbook? (@hnowlan ?)
The ideal would be to be able to only remove downtime that were set for a specific context in AlertManager.

Tue, Sep 22, 2:41 PM · SRE Observability
ayounsi lowered the priority of T403173: Automate PDU Deployment Process from Medium to Low.

I feel like as this is something not done at a high frequency, the priority to fully automate them will be lower than other projects. If that's the case the priority will depends on the complexity.

Tue, Sep 22, 9:53 AM · ops-eqiad, DC-Ops, Infrastructure-Foundations, SRE
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Tue, Sep 22, 8:38 AM · Infrastructure-Foundations, netops

Mon, Sep 21

ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Mon, Sep 21, 3:09 PM · Infrastructure-Foundations, netops
ayounsi renamed T437984: Junos file descriptors exhaustion from Junos file descriptors exhautions to Junos file descriptors exhaustion.
Mon, Sep 21, 3:09 PM · Infrastructure-Foundations, netops
ayounsi claimed T403173: Automate PDU Deployment Process.
Mon, Sep 21, 2:53 PM · ops-eqiad, DC-Ops, Infrastructure-Foundations, SRE
ayounsi closed T438291: Upgrade gNMIc to 0.49.0 as Resolved.

All done.

Mon, Sep 21, 8:23 AM · Infrastructure-Foundations, netops

Thu, Sep 17

ayounsi created T438291: Upgrade gNMIc to 0.49.0.
Thu, Sep 17, 7:23 AM · Infrastructure-Foundations, netops

Wed, Sep 16

ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Wed, Sep 16, 2:19 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Wed, Sep 16, 9:22 AM · Infrastructure-Foundations, netops

Tue, Sep 15

ayounsi moved T437984: Junos file descriptors exhaustion from Backlog to This quarter on the netops board.
Tue, Sep 15, 1:50 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Tue, Sep 15, 12:36 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Tue, Sep 15, 9:42 AM · Infrastructure-Foundations, netops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Tue, Sep 15, 8:47 AM · Infrastructure-Foundations, netops
ayounsi added a comment to T437984: Junos file descriptors exhaustion.

Deployed the workaround on asw1-b3-magru (not showing signs of issues yet)

Tue, Sep 15, 8:28 AM · Infrastructure-Foundations, netops
ayounsi updated the task description for T437984: Junos file descriptors exhaustion.
Tue, Sep 15, 8:18 AM · Infrastructure-Foundations, netops
ayounsi created T437984: Junos file descriptors exhaustion.
Tue, Sep 15, 8:17 AM · Infrastructure-Foundations, netops

Mon, Sep 14

ayounsi added a comment to T437024: Upgrade bgpalerter to 2.0.1.

The most likely fix to move forward is to move the bgpalerter instance on the netmon hosts as they have direct internet connectivity.

Mon, Sep 14, 2:28 PM · Infrastructure-Foundations

Tue, Sep 8

ayounsi added a comment to T390052: Enable gNMI on SRX devices and fasw.

Junos recommended for the SRX300 is now 24.4 so hopefully next year for 25.4, unless we decide to upgrade sooner.

Tue, Sep 8, 1:04 PM · Patch-For-Review, Infrastructure-Foundations, netops

Mon, Sep 7

ayounsi closed T424478: Upgrade netflow hosts to Trixie as Resolved.
Mon, Sep 7, 11:38 AM · Infrastructure-Foundations, netops

Sep 4 2026

ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 4 2026, 2:03 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 4 2026, 12:56 PM · Infrastructure-Foundations, netops
ayounsi updated subscribers of T437024: Upgrade bgpalerter to 2.0.1.
Sep 4 2026, 11:58 AM · Infrastructure-Foundations
ayounsi added a comment to T437024: Upgrade bgpalerter to 2.0.1.

There is unfortunately a catch that's going to warrant larger discussions: 2.0.0 removed proxy support, on which we rely :(

Sep 4 2026, 11:57 AM · Infrastructure-Foundations
ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 4 2026, 9:15 AM · Infrastructure-Foundations, netops

Sep 3 2026

ayounsi closed T436517: Create a 3rd Netflow instance in eqiad/codfw as Resolved.

All done.

Sep 3 2026, 3:48 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 3 2026, 3:45 PM · Infrastructure-Foundations, netops
ayounsi added a comment to T436898: LibreNMS PHP requirement.

Are you currently experiencing any issues with LibreNMS??

Sep 3 2026, 3:43 PM · observability
ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 3 2026, 1:23 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 3 2026, 1:23 PM · Infrastructure-Foundations, netops
ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 3 2026, 9:14 AM · Infrastructure-Foundations, netops
ayounsi created T436898: LibreNMS PHP requirement.
Sep 3 2026, 7:29 AM · observability
ayounsi renamed T371889: Upgrade Netbox to 4.7.x from Upgrade Netbox to 4.6.x to Upgrade Netbox to 4.7.x.
Sep 3 2026, 5:36 AM · Infrastructure-Foundations, netbox

Sep 2 2026

ayounsi updated the task description for T424478: Upgrade netflow hosts to Trixie.
Sep 2 2026, 12:46 PM · Infrastructure-Foundations, netops