Page MenuHomePhabricator

ayounsi (Arzhel Younsi)
Network Engineer

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Apr 3 2017, 6:23 PM (98 w, 3 d)
Availability
Available
IRC Nick
xionox
LDAP User
Ayounsi
MediaWiki User
AYounsi (WMF) [ Global Accounts ]

Recent Activity

Today

ayounsi added a comment to T205897: Netbox: fill network topology.

I imported all the cables except the servers' uplinks, see:
https://netbox.wikimedia.org/dcim/cables/?page=6 (and previous pages)

Fri, Feb 22, 8:28 AM · Operations
ayounsi updated the task description for T205897: Netbox: fill network topology.
Fri, Feb 22, 8:25 AM · Operations
ayounsi updated the task description for T205897: Netbox: fill network topology.
Fri, Feb 22, 8:24 AM · Operations
ayounsi added a comment to T215231: rack/setup/install labsdb1012.eqiad.wmnet.

Talking to @elukey about that.
In eqiad only rows A and C have the cloud-support vlan: https://netbox.wikimedia.org/ipam/vlans/?q=cloud-support&site=eqiad

Fri, Feb 22, 7:10 AM · Patch-For-Review, ops-eqiad, Analytics, User-Elukey, Operations
ayounsi reassigned T215231: rack/setup/install labsdb1012.eqiad.wmnet from ayounsi to Cmjohnson.

Talked to Chris, the server is actually in B8

Fri, Feb 22, 5:38 AM · Patch-For-Review, ops-eqiad, Analytics, User-Elukey, Operations
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Fri, Feb 22, 4:33 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Fri, Feb 22, 4:25 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi reassigned T216716: cp5007 correctable mem errors from ayounsi to RobH.
Fri, Feb 22, 2:14 AM · Operations, ops-eqsin, Traffic
ayounsi reassigned T216717: cp5006 correctable mem errors from ayounsi to RobH.
Fri, Feb 22, 2:14 AM · ops-eqsin, Traffic, Operations
ayounsi added a comment to T216716: cp5007 correctable mem errors.

Swapped A1 with A2 and A4 with A5

Fri, Feb 22, 2:06 AM · Operations, ops-eqsin, Traffic
ayounsi added a comment to T216717: cp5006 correctable mem errors.

Swapped A3 and A4

Fri, Feb 22, 1:52 AM · ops-eqsin, Traffic, Operations

Yesterday

ayounsi triaged T216691: amber light on cp5006/5007 as High priority.
Thu, Feb 21, 9:58 AM · Traffic, ops-eqsin, Operations
ayounsi added a comment to T213121: Deploy cr2-eqsin.

Also audited all the spares we have onsite:
https://docs.google.com/spreadsheets/d/1FKYVQJePjTQ7nVwYv4oDC6Gszk7RLrkq5ySN0fjvSoY/edit#gid=2057953856
Labelled the few cables not labelled:
https://docs.google.com/spreadsheets/d/1FKYVQJePjTQ7nVwYv4oDC6Gszk7RLrkq5ySN0fjvSoY/edit#gid=1665726692

Thu, Feb 21, 9:47 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi added a comment to T214274: Degraded RAID on cp5010.

return shipment ticket 1-185737841426 opened with Equinix, DHL should pick up the defective disk in the next few days.

Thu, Feb 21, 9:40 AM · Traffic, ops-eqsin, Operations
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 21, 7:24 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 21, 6:50 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 21, 6:26 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 21, 5:36 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 21, 4:22 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 21, 4:16 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops

Wed, Feb 20

ayounsi added a comment to T216469: Netbox: cable termination names report.

No strong preferences, but indeed "Device X has Y miss-labelled" is an option.

Wed, Feb 20, 5:12 AM · Operations-Software-Development, Operations
ayounsi added a comment to T180179: Evaluate the possibility to add Juniper images to Openstack.

@aborrero Being able to create different L2 links between VMs would be ideal, but having them all in the same L2 domain and able to talk to each other would already be a good start.

Wed, Feb 20, 5:10 AM · cloud-services-team (Kanban), Cloud-VPS, netops, Traffic, Operations

Tue, Feb 19

ayounsi added a comment to T204281: Stop prioritizing peering over transit.

This caused a ~80Mbps traffic drop on the peering link.

Tue, Feb 19, 6:57 AM · Performance-Team (Radar), netops, Operations
ayounsi claimed T190090: Offload pings to dedicated server.

Discussed it with Faidon,

Tue, Feb 19, 6:04 AM · Patch-For-Review, netops, Traffic, Operations
ayounsi added a comment to T204281: Stop prioritizing peering over transit.

Doing the change in ulsfo:

[edit policy-options policy-statement BGP_community_actions]
-    term peer-private-peer {
-        from community PEER_PRIVATE_PEER;
-        then {
-            local-preference 250;
-            next policy;
-        }
-    }
-    term peer {
-        from community PEERING_ROUTE;
-        then {
-            local-preference 250;
-            next policy;
-        }
-    }

For reference, here are all transit ports stacked:
https://librenms.wikimedia.org/graphs/id=7219,16765,7159/type=multiport_bits_separate/
And here is peering:
https://librenms.wikimedia.org/graphs/id=16787/type=port_bits/

Tue, Feb 19, 5:30 AM · Performance-Team (Radar), netops, Operations
ayounsi triaged T216469: Netbox: cable termination names report as Lowest priority.
Tue, Feb 19, 3:26 AM · Operations-Software-Development, Operations
ayounsi added a comment to T205897: Netbox: fill network topology.

So, I don't think we can reasonably expect our on-site techs to look at a box and say "oh this port is enp4s0f0p1" and record it as such :)

Tue, Feb 19, 2:55 AM · Operations
ayounsi reassigned T191362: decom promethium/WMF3571 from ayounsi to Andrew.

This is because ge-3/0/19 is the last interface in the interface-range vlan-cloud-instances1-b-eqiad.

Tue, Feb 19, 12:40 AM · decommission, Operations, DC-Ops, ops-eqiad

Mon, Feb 18

ayounsi added a comment to T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
[edit firewall family inet filter cloud-in4]
       term labsdb { ... }
+      /* T216353 */
+      term clouddb_return {
+          from {
+              destination-address {
+                  /* labsdb1004, labsdb1005 */
+                  10.64.37.8/31;
+                  /* labsdb1006 */
+                  10.64.37.11/32;
+                  /* labsdb1007 */
+                  10.64.37.12/32;
+                  /* dbproxy1010, dbproxy1011 */
+                  10.64.37.14/31;
+                  /* labstore1005 */
+                  10.64.37.20/32;
+                  /* nfs-tools, labstore1004 */
+                  10.64.37.18/31;
+              }
+              protocol tcp;
+              source-port 3306;
+          }
+          then accept;
+      }                                
       term labmon--http { ... }
[edit firewall family inet filter cloud-in4]
-      /* T216353 */
-      term labsdb_return {
-          from {
-              destination-address {
-                  /* labsdb1004, labsdb1005 */
-                  10.64.37.8/31;
-                  /* labsdb1006 */
-                  10.64.37.11/32;
-                  /* labsdb1007 */
-                  10.64.37.12/32;
-                  /* dbproxy1010, dbproxy1011 */
-                  10.64.37.14/31;
-              }
-              protocol tcp;
-              source-port 3306;
-          }
-          then accept;
-      }
Mon, Feb 18, 12:27 PM · User-Marostegui, netops, Operations, cloud-services-team (Kanban), Cloud-VPS

Sun, Feb 17

ayounsi added a comment to T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).
[edit firewall family inet filter cloud-in4]
       term labsdb { ... }
+      /* T216353 */
+      term labsdb_return {
+          from {
+              destination-address {
+                  /* labsdb1004, labsdb1005 */
+                  10.64.37.8/31;
+                  /* labsdb1006 */
+                  10.64.37.11/32;
+                  /* labsdb1007 */
+                  10.64.37.12/32;
+                  /* dbproxy1010, dbproxy1011 */
+                  10.64.37.14/31;
+              }
+              protocol tcp;
+              source-port 3306;
+          }
+          then accept;
+      }
       term labmon--http { ... }
Sun, Feb 17, 1:13 PM · User-Marostegui, netops, Operations, cloud-services-team (Kanban), Cloud-VPS

Thu, Feb 14

ayounsi changed the status of T180179: Evaluate the possibility to add Juniper images to Openstack from Stalled to Open.

Bumping this task, now that WMCS is on Neutron.

Thu, Feb 14, 6:02 AM · cloud-services-team (Kanban), Cloud-VPS, netops, Traffic, Operations
ayounsi added a comment to T215193: Fix codfw x-connect 65373.

Ticket opened with CyrusOne.

Thu, Feb 14, 5:43 AM · Operations, netops
ayounsi updated subscribers of T213121: Deploy cr2-eqsin.

@BBlack @faidon @mark reviews welcome if you have some time!

Thu, Feb 14, 3:57 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 14, 3:57 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T213121: Deploy cr2-eqsin.
Thu, Feb 14, 3:42 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi claimed T213121: Deploy cr2-eqsin.
Thu, Feb 14, 1:17 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops

Wed, Feb 13

ayounsi added a comment to T205897: Netbox: fill network topology.

I started to format the spreadsheat in a way that can be imported in Netbox:
https://docs.google.com/spreadsheets/d/1FKYVQJePjTQ7nVwYv4oDC6Gszk7RLrkq5ySN0fjvSoY/edit#gid=1665726692
The plan is to then delete the existing cables and import them again with all their attributes.

Wed, Feb 13, 9:16 AM · Operations
ayounsi updated the task description for T174616: set up cr3-esams.
Wed, Feb 13, 2:44 AM · ops-esams, Operations, netops
ayounsi added a subtask for T215991: Repurpose csw2-oe14/15 and lab-ex4200 as msw: T184065: Setup new access switches.
Wed, Feb 13, 2:35 AM · ops-esams, Operations
ayounsi added a parent task for T184065: Setup new access switches: T215991: Repurpose csw2-oe14/15 and lab-ex4200 as msw.
Wed, Feb 13, 2:35 AM · Operations, ops-esams
ayounsi triaged T215991: Repurpose csw2-oe14/15 and lab-ex4200 as msw as Normal priority.
Wed, Feb 13, 2:34 AM · ops-esams, Operations
ayounsi updated the task description for T184065: Setup new access switches.
Wed, Feb 13, 2:30 AM · Operations, ops-esams
ayounsi closed T150264: Icinga check for VRRP as Resolved.

Confirmed working: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=vrrp

Wed, Feb 13, 12:43 AM · Patch-For-Review, netops, monitoring, Operations

Wed, Feb 6

ayounsi added a comment to T213121: Deploy cr2-eqsin.
In T205487#4932700, @Cmjohnson wrote:

@ayounsi This is the contents that I am shipping to eqsin. Please confirm that is all you need

12 SFP-10GLR Transceivers
(2) 3M LC-LC fiber
(4) 5M LC-LC fiber
(3) 5M LC-SC fiber

(1) mx204
(2) c13-c14 power cables

Confirmed (+mounting brackets or rails)
I'm pretty sure we got spares, but can you add 1 console cable, and 1 cat5 (mgmt), 1m for both? just in case.

Wed, Feb 6, 7:22 PM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi added a comment to T212273: Spike of multicast traffic.

My guess so far is that the recabling triggered a bug in Junos VCF which caused a multicast storm that got propagated to all listeners, filling up links and exhausting resources.
After some research, there are some knobs we can tune to:
1/ Prevent router's exhaustion by applying more strict DDoS thresholds
2/ Prevent links saturation by applying Multicast Bandwidth Maximums or firewall policer on key interfaces (access/core and core-core links).
Eyeballing the graphs, 300Mbps seems like a conservative value to use, fine tuning it would be ideal.

Wed, Feb 6, 6:03 PM · Operations, netops
ayounsi claimed T215364: Deploy pfw policy to allow https to frmon.frdev.wikimedia.org.

Done. No diff in codfw, 1 new rule in eqiad.

Wed, Feb 6, 2:33 AM · netops, Operations, fundraising-tech-ops

Tue, Feb 5

ayounsi added a comment to T213128: Replace eqiad mgmt switches with EX4200s.

This is fine, I only need 1 for tests, once in prod they can do without.

Tue, Feb 5, 1:20 AM · ops-eqiad, Operations, netops

Mon, Feb 4

ayounsi reassigned T215173: Clean labnodepool1001.eqiad.wmnet from firewall / router from ayounsi to hashar.

No mention of that IP in Rancid (router configs).

Mon, Feb 4, 8:16 PM · cloud-services-team (Kanban), Continuous-Integration-Infrastructure (shipyard)
ayounsi triaged T215193: Fix codfw x-connect 65373 as Normal priority.
Mon, Feb 4, 7:30 PM · Operations, netops

Sat, Jan 26

ayounsi reassigned T213121: Deploy cr2-eqsin from ayounsi to Cmjohnson.
Sat, Jan 26, 12:26 AM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops

Fri, Jan 25

ayounsi claimed T213121: Deploy cr2-eqsin.
Fri, Jan 25, 8:34 PM · Patch-For-Review, ops-eqiad, ops-eqsin, Operations, netops
ayounsi updated the task description for T205897: Netbox: fill network topology.
Fri, Jan 25, 5:23 PM · Operations
ayounsi added a comment to T205897: Netbox: fill network topology.

I looked into that and imported all the cables related to scs-eqsin to test the new feature:

  • The cable list doesn't allow to filter/sort by endpoint. Which makes mass editing a pain. As we can't easily do "all cables connected to ps1 are black and power". Should be reported upstream.
  • Adding cable ID via the UI requires a lot of clicks.
  • "Power" is the only option, in the dropdown, we can't specify the connector type. Upstream task.
Fri, Jan 25, 5:22 PM · Operations

Thu, Jan 24

ayounsi closed T214459: Connection problem (Moscow ISP, 4G) with Beeline / Sovintel as Resolved.

Good to see a happy ending here. Thanks to everyone who helped.

Thu, Jan 24, 11:32 PM · Traffic, Operations, netops
ayounsi added a comment to T214459: Connection problem (Moscow ISP, 4G) with Beeline / Sovintel.
Thu, Jan 24, 8:17 PM · Traffic, Operations, netops

Wed, Jan 23

ayounsi added a comment to T214459: Connection problem (Moscow ISP, 4G) with Beeline / Sovintel.

I think the "poliplastic" track a red herring, even though those few IPs are assigned to them in whois, it's used by AS3216, AS16345's upstream, most likely as router IPs.
The fact that the traceroute from us to 83.220.238.125 stops is probably due to the NAT gateway (or something on the way) blocking ICMP.
The fact that pings and traceroutes from them to us works, indicates that routing is most likely fine.

Wed, Jan 23, 6:58 PM · Traffic, Operations, netops
ayounsi added a comment to T184063: Remove all decommissioned hardware.

Removed:

maerlant
multatuli
eeden
nescio

from the list as they are still in Puppet (and some it prod).

Wed, Jan 23, 5:13 PM · Operations, Epic, ops-esams
ayounsi updated the task description for T184063: Remove all decommissioned hardware.
Wed, Jan 23, 5:11 PM · Operations, Epic, ops-esams
ayounsi claimed T211730: Replace accepted-prefix-limit with prefix-limit.
Wed, Jan 23, 3:57 PM · netops, Operations

Jan 22 2019

ayounsi added a comment to T204281: Stop prioritizing peering over transit.

First observation shows a ~6/800Mbps traffic shift from peering to transit, which is small and within expected range.

Jan 22 2019, 5:19 PM · Performance-Team (Radar), netops, Operations
ayounsi added a comment to T204281: Stop prioritizing peering over transit.

For reference, here are all the Amsterdam transit ports stacked:
https://librenms.wikimedia.org/graphs/id=5944,11510,4109,4110,4151,6862/type=multiport_bits_separate/
And here is peering:
https://librenms.wikimedia.org/graphs/id=11501/type=port_bits/

Jan 22 2019, 4:03 PM · Performance-Team (Radar), netops, Operations

Jan 17 2019

ayounsi reassigned T200277: OSPF metrics from ayounsi to faidon.
Jan 17 2019, 10:03 PM · netops, Operations
ayounsi lowered the priority of T200277: OSPF metrics from Normal to Low.

Low priority, over to @faidon for feedbacks.

Jan 17 2019, 10:03 PM · netops, Operations
ayounsi closed T209145: Investigate network issues in codfw that caused 503 errors as Resolved.

My guess based on those clues, is that this link flap caused at least some traffic from eqiad to codfw to be blackholed. Most likely the time protocols re-converge properly.
The DDOS_PROTOCOL_VIOLATION Rejectv6:aggregate log means that traffic spiked toward a v6 aggregate prefix, that didn't have more specific prefixes.
Most likely 2620:0:860::/48 (codfw) disappeared from 2620:0:860::/46. And legit traffic triggered that log.
If that's correct, there are some suggestions to make such events less impactful, such as T167306.

Jan 17 2019, 10:02 PM · Operations, netops
ayounsi reassigned T211728: Outbound BGP graceful shutdown from ayounsi to faidon.

Over to @faidon for review/feedback.

Jan 17 2019, 7:25 PM · Operations, netops
ayounsi updated the task description for T211728: Outbound BGP graceful shutdown.
Jan 17 2019, 7:24 PM · Operations, netops
ayounsi added a parent task for T189552: Rack/cable/configure ulsfo MX204: T196432: Configure interface damping on primary links.
Jan 17 2019, 7:17 PM · Patch-For-Review, Operations, ops-ulsfo, netops, Traffic
ayounsi added subtasks for T196432: Configure interface damping on primary links: T189552: Rack/cable/configure ulsfo MX204, T174616: set up cr3-esams.
Jan 17 2019, 7:17 PM · Operations, Traffic, netops
ayounsi added a parent task for T174616: set up cr3-esams: T196432: Configure interface damping on primary links.
Jan 17 2019, 7:17 PM · ops-esams, Operations, netops
ayounsi claimed T211254: Free up 185.15.59.0/24.

In addition to the above DNS change, the following needs to change on the routers:

Jan 17 2019, 7:13 PM · Patch-For-Review, Traffic, netops, Operations
ayounsi reassigned T190090: Offload pings to dedicated server from ayounsi to faidon.

Discussed it with Brandon and we think that option 3 is the best path forward. Over to @faidon for thoughts/review.

Jan 17 2019, 6:18 PM · Patch-For-Review, netops, Traffic, Operations
ayounsi lowered the priority of T201039: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46 from Normal to Low.

Discussed it with Brandon, it's still something we want to fix but is now low priority.
We will probably have to wait for the next DC failover or another more urgent reason to upgrade.

Jan 17 2019, 4:56 PM · Patch-For-Review, Operations, netops
ayounsi reassigned T211730: Replace accepted-prefix-limit with prefix-limit from ayounsi to faidon.

Over to Faidon for feedback.

Jan 17 2019, 12:01 AM · netops, Operations

Jan 16 2019

ayounsi added a subtask for T207753: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22: T211254: Free up 185.15.59.0/24.
Jan 16 2019, 11:56 PM · netops, Operations
ayounsi added a parent task for T211254: Free up 185.15.59.0/24: T207753: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22.
Jan 16 2019, 11:56 PM · Patch-For-Review, Traffic, netops, Operations
ayounsi added a parent task for T211254: Free up 185.15.59.0/24: Unknown Object (Task).
Jan 16 2019, 11:53 PM · Patch-For-Review, Traffic, netops, Operations
ayounsi added a comment to T201039: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46.

Reducing priority as the situation is stable.
At this point I don't think the cost of upgrading the switch stacks of row B and C (full row down for ~15min) is worth the advantage of enabling IGMP snooping.

Jan 16 2019, 11:46 PM · Patch-For-Review, Operations, netops
ayounsi renamed T201039: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46 from connectivity issues between several hosts on asw2-b-eqiad to IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46.
Jan 16 2019, 11:37 PM · Patch-For-Review, Operations, netops
ayounsi reassigned T189522: Detect IP address collisions from ayounsi to faidon.

Note that I also created an LibreNMS alert to monitor explicitly the mgmt network:
%syslog.msg ~ "KERN_ARP_ADDR_CHANGE" && %devices.hostname ~ "mr" && %devices.hardware ~ "SRX" && %syslog.timestamp >= %macros.past_30m
mgmt only as "KERN_ARP_ADDR_CHANGE" is only for SRX.

Jan 16 2019, 11:32 PM · Patch-For-Review, Operations, netops
ayounsi moved T184293: rack/setup/install lvs101[3-6] from Blocked to Backlog on the ops-eqiad board.
Jan 16 2019, 10:14 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
ayounsi updated the task description for T184293: rack/setup/install lvs101[3-6].
Jan 16 2019, 10:13 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
ayounsi added a comment to T190090: Offload pings to dedicated server.

Diff for option 3 in eqiad is:

Jan 16 2019, 7:09 PM · Patch-For-Review, netops, Traffic, Operations

Jan 15 2019

ayounsi added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

I think there is a distinction to make here when saying "prod", as it's made of several vlans/networks, especially:

  • public, host have public IPs and are reachable from the Internet (including Cloud), protected by iptables
  • private, host have private IPs, only reachable from private and prod public, protected by firewalls and iptables
  • analytics, similar to private, with firewalls limiting outbound flows as well (data leak, etc)
Jan 15 2019, 11:32 PM · Analytics-Kanban, netops, Operations, Analytics
ayounsi closed T212791: Interface errors on cr1-eqiad:xe-3/3/1 as Resolved.
Jan 15 2019, 10:49 PM · Operations, ops-eqiad
ayounsi added a comment to T212791: Interface errors on cr1-eqiad:xe-3/3/1.

Zayo tech swapped their optic and so far no more errors.

Jan 15 2019, 9:51 PM · Operations, ops-eqiad
ayounsi added a comment to T212791: Interface errors on cr1-eqiad:xe-3/3/1.

Told Zayo about Equinix's test/cleanup of the X-connect. Then received a "Dispatch Charge Notification" without approval request, and the folloing Zayo updates:

"Good afternoon, we received your update and appreciate the information. We will continue to investigate this issue and provide you with our findings."

Jan 15 2019, 9:20 PM · Operations, ops-eqiad
ayounsi added a comment to T204281: Stop prioritizing peering over transit.

Scheduling the work on Tuesday January 22nd, 16:00UTC, scope is Amsterdam only.
That gives us the remaining of the week to monitor for any issue. Then collect/analyze results after the All Hands.

Jan 15 2019, 5:30 PM · Performance-Team (Radar), netops, Operations
ayounsi assigned T213790: mgmt host interfaces down for rack D7 in codfw due to ge-0/0/30 on msw1-codfw down to Papaul.

Papaul, can you please verify the status of msw-d7-codfw and its link to msw1-codfw, and replace any faulty part if necessary.

Jan 15 2019, 4:03 PM · netops, Operations

Jan 14 2019

ayounsi updated the task description for T204281: Stop prioritizing peering over transit.
Jan 14 2019, 11:35 PM · Performance-Team (Radar), netops, Operations
ayounsi added a comment to T212791: Interface errors on cr1-eqiad:xe-3/3/1.

Equinix cleaned and tested the X-connect, but the issue persists.
Next step is to do another round of testing/swapping on our side and follow up with Zayo if no resolution.

Jan 14 2019, 10:20 PM · Operations, ops-eqiad

Jan 9 2019

ayounsi added a comment to T204281: Stop prioritizing peering over transit.

The following need to be pushed to routers to stop adding a higher local_pref to routes learned via peering (IXP+private).

[edit policy-options policy-statement BGP_community_actions]
!     inactive: term peer-private-peer { ... }
!     inactive: term peer { ... }

If successful those terms could be deleted instead of deactivated.

Jan 9 2019, 11:56 PM · Performance-Team (Radar), netops, Operations
ayounsi reassigned T211930: Add eqsin routing special cases to jnt from ayounsi to faidon.

Thanks for the feedback! If the back and forth in diffs via a task gets old, we can find a different solution.

Jan 9 2019, 11:20 PM · Operations, netops
ayounsi reassigned T150264: Icinga check for VRRP from ayounsi to faidon.

CR tested and updated, ready for reviews.

Jan 9 2019, 4:35 PM · Patch-For-Review, netops, monitoring, Operations
ayounsi updated the task description for T213233: asw-c-codfw - FPC 1 PEM 1 is not powered.
Jan 9 2019, 12:02 AM · ops-codfw, Operations

Jan 8 2019

ayounsi triaged T213233: asw-c-codfw - FPC 1 PEM 1 is not powered as High priority.
Jan 8 2019, 11:59 PM · ops-codfw, Operations
ayounsi added a comment to T212791: Interface errors on cr1-eqiad:xe-3/3/1.

Reply from Zayo:

Jan 8 2019, 11:31 PM · Operations, ops-eqiad
ayounsi updated the task description for T212348: Move servers off asw2-a5-eqiad.
Jan 8 2019, 5:38 PM · ops-eqiad, netops, Operations
ayounsi added a comment to T207965: eqiad: Re-connect cage cameras .

I created T213128 for using EX4200 as mgmt switches. Even though it would be the best option in the long term, it comes with its own challenges.

Jan 8 2019, 12:14 AM · Operations, ops-eqiad
ayounsi triaged T213128: Replace eqiad mgmt switches with EX4200s as Normal priority.
Jan 8 2019, 12:01 AM · ops-eqiad, Operations, netops

Jan 7 2019

ayounsi added a parent task for T187960: Rack/cable/configure asw2-a-eqiad switch stack: T213122: Increase network capacity (2018-19 Q3 Goal).
Jan 7 2019, 11:00 PM · Patch-For-Review, Operations, ops-eqiad, netops