Page MenuHomePhabricator

ayounsi (Arzhel Younsi)
Network Engineer

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Apr 3 2017, 6:23 PM (106 w, 6 d)
Availability
Available
IRC Nick
xionox
LDAP User
Ayounsi
MediaWiki User
AYounsi (WMF) [ Global Accounts ]

Recent Activity

Sat, Apr 20

ayounsi added a comment to T213843: Juniper network device audit - all sites.

yep! Juniper has been working on it since a few weeks in ticket: 2019-0408-0694
Based on https://docs.google.com/spreadsheets/d/1tJ-mqN4-g_NyvO24pRERxVTbX6AMe6lMG9YcO2840Vg/edit (Tab Final)

Sat, Apr 20, 9:00 PM · DC-Ops, netops, Operations

Thu, Apr 18

ayounsi closed T221388: Test dhcp-option 82 as Resolved.
Thu, Apr 18, 4:12 PM · Operations, netops
ayounsi added a comment to T221259: eqord - ulsfo Telia link down - IC-313592.

Got an email 1h ago saying the onsite crew was still splicing hard.

Thu, Apr 18, 2:44 PM · Operations, netops
ayounsi closed Restricted Task, a subtask of T221274: configure and install frav1002.frack.eqiad.wmnet, as Resolved.
Thu, Apr 18, 2:43 PM · fundraising-tech-ops
ayounsi added a comment to T221156: cr4-ulsfo rebooted unexpectedly.

I have checked the core and the information, we did not find any PR related to this, please give us a few days to analyze the core.

Thu, Apr 18, 12:00 AM · Operations, netops

Wed, Apr 17

ayounsi added a comment to T221259: eqord - ulsfo Telia link down - IC-313592.

Indeed, just got a notification:
"We have an outage which is suspected to be caused by a cable fault. Our NOC is investigating and activating local resources. We will provide more information as it becomes available."

Wed, Apr 17, 4:40 PM · Operations, netops
ayounsi closed T221232: configure switch ports for frav1002.frack.eqiad.wmnet as Resolved.
Wed, Apr 17, 4:12 PM · Operations, netops, fundraising-tech-ops
ayounsi closed T221232: configure switch ports for frav1002.frack.eqiad.wmnet, a subtask of T213104: rack and cable frav1002.frack.eqiad.wmnet, as Resolved.
Wed, Apr 17, 4:12 PM · Patch-For-Review, Operations, fundraising-tech-ops, ops-eqiad
ayounsi triaged T221156: cr4-ulsfo rebooted unexpectedly as Normal priority.
Wed, Apr 17, 12:12 AM · Operations, netops
ayounsi claimed T221156: cr4-ulsfo rebooted unexpectedly.
Apr 16 23:20:49  cr4-ulsfo kernel: spin lock 0xfffff80012ce73c0 (turnstile lock) held by 0xfffff8000941d560 (tid 100012) too long
Apr 16 23:20:49  cr4-ulsfo kernel: panic: spin lock held too long
Apr 16 23:20:49  cr4-ulsfo kernel: cpuid = 0
Apr 16 23:20:49  cr4-ulsfo kernel: Uptime: 204d2h14m5s
Apr 16 23:20:49  cr4-ulsfo kernel: Dumping 1237 out of 16341 MB:..2%..11%..21%..32%..41%..51%..61%..72%..81%..91%
Apr 16 23:20:49  cr4-ulsfo kernel: Dump complete
Apr 16 23:20:49  cr4-ulsfo kernel: Automatic reboot in 15 seconds - press a key on the console to abort
Apr 16 23:20:49  cr4-ulsfo kernel: Rebooting...

And it generated a 1.2G core dump.

Wed, Apr 17, 12:12 AM · Operations, netops

Tue, Apr 16

ayounsi added a comment to T184293: rack/setup/install lvs101[3-6].

Vlan is configured for eth0 of those two servers, the port is still showing as down though.
I also configured the vlan for their eth1 port.

Tue, Apr 16, 10:20 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
ayounsi updated the task description for T184293: rack/setup/install lvs101[3-6].
Tue, Apr 16, 10:18 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
ayounsi updated the task description for T221142: Willy Pao onboarding.
Tue, Apr 16, 9:20 PM · Patch-For-Review, Operations, DC-Ops
ayounsi updated the task description for T221142: Willy Pao onboarding.
Tue, Apr 16, 9:11 PM · Patch-For-Review, Operations, DC-Ops
ayounsi updated the task description for T221142: Willy Pao onboarding.
Tue, Apr 16, 9:02 PM · Patch-For-Review, Operations, DC-Ops
ayounsi updated the task description for T221142: Willy Pao onboarding.
Tue, Apr 16, 9:02 PM · Patch-For-Review, Operations, DC-Ops
ayounsi triaged T221142: Willy Pao onboarding as High priority.
Tue, Apr 16, 8:39 PM · Patch-For-Review, Operations, DC-Ops

Mon, Apr 15

ayounsi closed T219384: allow bast2002 to connect to mgmt network as Resolved.

I think @RobH's comment was about the fact that bast hosts are not allowed to reach mgmt's http/https, but only ssh.
On the other hand, cumin hosts are allowed to.

Mon, Apr 15, 7:10 PM · netops, Operations
ayounsi closed T219384: allow bast2002 to connect to mgmt network, a subtask of T196665: rack/setup/install bast2002.wikimedia.org, as Resolved.
Mon, Apr 15, 7:10 PM · ops-codfw, Operations

Sat, Apr 13

ayounsi created T220887: Allow Bryan Davis to downtime alerts in Icinga.
Sat, Apr 13, 6:45 PM · Patch-For-Review, Operations, SRE-Access-Requests, monitoring

Fri, Apr 12

ayounsi created T220836: Guidelines for Rust/Go tools deployment.
Fri, Apr 12, 4:46 PM · Packaging, serviceops
ayounsi added a comment to T220700: Upgrade kafka-jumbo100[1-6] to 10G NICs (if possible).

From: https://netbox.wikimedia.org/dcim/devices/?q=kafka-jumbo&status=1
kafka-jumbo1002
kafka-jumbo1004
kafka-jumbo1005

Fri, Apr 12, 3:47 PM · netops, ops-eqiad, hardware-requests, Operations, Analytics, User-Elukey
ayounsi closed T220716: Juniper security advisories (April 2019) as Resolved.

thanks, tl;dr; all good!

Fri, Apr 12, 1:19 AM · Operations, netops

Thu, Apr 11

ayounsi triaged T220669: RPKI Validation as Normal priority.
Thu, Apr 11, 12:22 AM · Operations, netops

Wed, Apr 10

ayounsi created T220639: Show IPs matching a list of IP subnets in Webrequest data.
Wed, Apr 10, 5:46 PM · User-Elukey, Analytics

Mon, Apr 8

ayounsi closed T220081: Allow swift https access from analytics to prod, a subtask of T219544: Make hadoop cluster able to push to swift , as Resolved.
Mon, Apr 8, 5:43 PM · Research, Operations, Discovery, Analytics
ayounsi closed T220081: Allow swift https access from analytics to prod as Resolved.

Done, please reopen if any issue.

Mon, Apr 8, 5:43 PM · netops, Operations, Analytics

Fri, Apr 5

ayounsi claimed T220081: Allow swift https access from analytics to prod.
Fri, Apr 5, 5:16 PM · netops, Operations, Analytics
ayounsi added a comment to T166066: Integrate the puppet compiler in the puppet CI pipeline.

That's useful, thank you.

Fri, Apr 5, 12:29 AM · Puppet, puppet-compiler, Release-Engineering-Team (Watching / External), Operations

Thu, Apr 4

ayounsi added a comment to T106056: set up a looking glass for WMF ASes.

Note that we peer with RIPE RIS collectors in out POPs, so people can use https://stat.ripe.net/widget/looking-glass as a looking glass.

Thu, Apr 4, 9:50 PM · Patch-For-Review, Operations, netops

Wed, Apr 3

ayounsi triaged T220050: shell user conflict in cloud realm as Low priority.
Wed, Apr 3, 10:55 PM · cloud-services-team, LDAP
ayounsi renamed T219952: IP header IN errors on cloud networks from IP hardware IN errors on cloud networks to IP header IN errors on cloud networks.
Wed, Apr 3, 3:44 PM · cloud-services-team (Kanban)

Tue, Apr 2

ayounsi closed T211730: Replace accepted-prefix-limit with prefix-limit as Resolved.

All set, no down or bouncing peers, no mentions of accepted-prefix-limit in Rancid

Tue, Apr 2, 5:13 PM · Operations, netops
ayounsi closed T219847: Outage on the primary codfw-eqsin link as Resolved.

Telia stabilized the situation, " Services should be stable at the moment, hands are off and we are working with the vendor to provide an RFO in the next 3 to 5 business days"

Tue, Apr 2, 3:43 PM · Operations, netops, Traffic
ayounsi updated the task description for T219847: Outage on the primary codfw-eqsin link.
Tue, Apr 2, 1:15 AM · Operations, netops, Traffic
ayounsi triaged T219847: Outage on the primary codfw-eqsin link as Normal priority.
Tue, Apr 2, 1:00 AM · Operations, netops, Traffic
ayounsi added a comment to T211730: Replace accepted-prefix-limit with prefix-limit.

Confirmed that replacing accepted-prefix-limit with prefix-limit does NOT cause the peer to bounce.

Tue, Apr 2, 12:21 AM · Operations, netops

Mon, Apr 1

ayounsi closed T211930: Add eqsin routing special cases to jnt as Resolved.

Pushed progressively and confirmed with the looking glasses that only the proper communities were received on the other side.
As well as the proper local_pref was applied.
Cleaned up the old policies and pushed everything to jnt.

Mon, Apr 1, 11:01 PM · Operations, netops
ayounsi added a comment to T211930: Add eqsin routing special cases to jnt.
cr2-eqsin
[edit protocols bgp group Transit4]
-    import [ BGP_sanitize_in BGP_transit_in BGP_avoid_long_RTT_in BGP_community_actions ];
+    import [ BGP_sanitize_in BGP_transit_in BGP_community_actions ];
-    export [ BGP_avoid_long_RTT_out BGP_outfilter ];
+    export BGP_outfilter;
[edit protocols bgp group Transit4 neighbor 180.87.164.61]
+     import [ BGP_sanitize_in BGP_transit_in AS6453_in BGP_community_actions ];
+     export [ AS6453_out BGP_outfilter ];
[edit protocols bgp group Transit4 neighbor 116.51.26.209]
+     import [ BGP_sanitize_in BGP_transit_in AS2914_in BGP_community_actions ];
+     export [ AS2914_out BGP_outfilter ];
[edit protocols bgp group Private-Peer4 neighbor 103.102.166.135]
+     import [ BGP_sanitize_in BGP_Private_Peer_in AS3491_in BGP_community_actions ];
Mon, Apr 1, 10:21 PM · Operations, netops
ayounsi added a comment to T211930: Add eqsin routing special cases to jnt.
cr1-eqsin
[edit protocols bgp group Transit4]
-    import [ BGP_sanitize_in BGP_transit_in BGP_avoid_long_RTT_in BGP_community_actions ];
+    import [ BGP_sanitize_in BGP_transit_in BGP_community_actions ];
-    export [ BGP_avoid_long_RTT_out BGP_outfilter ];
+    export BGP_outfilter;
[edit protocols bgp group Transit4 neighbor 62.115.148.76]
+     import [ BGP_sanitize_in BGP_transit_in AS1299_in BGP_community_actions ];
+     export [ AS1299_out BGP6_outfilter ];
Mon, Apr 1, 9:28 PM · Operations, netops
ayounsi closed T219591: ulsfo <-> codfw transit link flapping causing nginx availability alerts as Resolved.

Link has been up for 1+ day. Got a notification saying the emergency maintenance was done.

Mon, Apr 1, 7:01 PM · Patch-For-Review, Traffic, Operations, netops
ayounsi closed T218059: asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1 flapping as Resolved.

There was a VC link between FPC3 and FPC8 that was acting up/flapping long time ago.
As we already had quite too many VC links, I disabled as well but left it connected in case we needed to turn it on quicky.
Those logs were the fallouts of that, where a cable was plugged between the two members but not configured.

Mon, Apr 1, 6:15 PM · Operations, ops-eqiad

Fri, Mar 29

ayounsi claimed T219591: ulsfo <-> codfw transit link flapping causing nginx availability alerts.
Fri, Mar 29, 3:52 PM · Patch-For-Review, Traffic, Operations, netops

Thu, Mar 28

ayounsi added a comment to T211930: Add eqsin routing special cases to jnt.

Done for IPv6. Confirmed that we apply the proper local-pref, and we advertise the proper communities to the proper peers.
I'll do v4 after the weekend.

Thu, Mar 28, 11:01 PM · Operations, netops
ayounsi added a comment to T219486: Send peering requests to AS with the worst TTFB.

(Added you to the task)

Thu, Mar 28, 5:25 PM · Traffic, Performance-Team, Operations
ayounsi claimed T211930: Add eqsin routing special cases to jnt.

Moving forward on that as the latest plan (taking the feedback into consideration) is anyway better than what we currently have deployed in Singapore.

Thu, Mar 28, 4:49 PM · Operations, netops
ayounsi added a comment to T219486: Send peering requests to AS with the worst TTFB.

The first step when looking at peering with a provider is to check if we're both present at a common exchange point.
You can see where we are present on https://www.peeringdb.com/net/1365, especially the "Public Peering Exchange Points" list.

Thu, Mar 28, 4:00 PM · Traffic, Performance-Team, Operations

Wed, Mar 27

ayounsi claimed T219384: allow bast2002 to connect to mgmt network.
Wed, Mar 27, 4:28 PM · netops, Operations

Mon, Mar 25

ayounsi closed T190090: Offload pings to dedicated server as Resolved.

Everything needed here is done.
Full doc on https://wikitech.wikimedia.org/wiki/Ping_offload
Will open a followup task once the Ganeti clusters are ready in the POPs T96852

Mon, Mar 25, 9:47 PM · Patch-For-Review, netops, Traffic, Operations
ayounsi updated the task description for T190090: Offload pings to dedicated server.
Mon, Mar 25, 9:46 PM · Patch-For-Review, netops, Traffic, Operations
ayounsi closed T218307: eqiad - eqord Telia link down - IC-314533 as Resolved.

Link is back up. Cf. email thread for full details. But tl;dr; outdated cable label caused onsite tech to unplug our link.

Mon, Mar 25, 9:42 PM · Operations, netops

Mar 20 2019

ayounsi added a comment to T190090: Offload pings to dedicated server.

Next step is to apply the following to replace the test IP with codfw text-lb IP.

[edit firewall family inet filter transport-in4 term no-offload-ping4 from destination-address]
+        208.80.153.224/32;
-        208.80.153.225/32;
[edit firewall family inet filter transport-in4 term offload-ping4 from destination-address]
+        208.80.153.224/32;
-        208.80.153.225/32;
[edit firewall family inet filter border-in4 term offload-ping4 from destination-address]
+        208.80.153.224/32;
-        208.80.153.225/32;
Mar 20 2019, 9:51 PM · Patch-For-Review, netops, Traffic, Operations
ayounsi added a comment to T190090: Offload pings to dedicated server.
cr2-codfw
[edit interfaces xe-5/0/0]
-   description "Core: cr2-eqdfw:xe-0/1/4 (CyrusOne wikimedia:ix2.dfw4_to_ix2.dfw5.245.0009) {#11403} [10Gbps wave]";
+   description "Transport: cr2-eqdfw:xe-0/1/4 (CyrusOne wikimedia:ix2.dfw4_to_ix2.dfw5.245.0009) {#11403} [10Gbps wave]";
[edit interfaces xe-5/0/0 unit 0 family inet]
+       filter {
+           input transport-in4;
+       }
[edit interfaces xe-5/0/1 unit 0 family inet]
+       filter {
+           input transport-in4;
+       }
[edit interfaces xe-5/2/1 unit 0 family inet]
+       filter {
+           input transport-in4;
+       }
[edit firewall family inet]
+     /* T190090 */
+     filter transport-in4 {
+         term no-offload-ping4 {
+             from {
+                 destination-address {
+                     208.80.153.225/32;
+                 }
+                 source-prefix-list {
+                     wikimedia4;
+                     trusted-space4;
+                 }
+                 protocol icmp;
+                 icmp-type echo-request;
+             }                         
+             then accept;
+         }
+         term offload-ping4 {
+             from {
+                 destination-address {
+                     208.80.153.225/32;
+                 }
+                 protocol icmp;
+                 icmp-type echo-request;
+             }
+             then {
+                 next-ip 10.192.0.22/32;
+             }
+         }
+         term default {
+             then accept;
+         }
+     }
      filter border-in4 { ... }
Mar 20 2019, 9:29 PM · Patch-For-Review, netops, Traffic, Operations
ayounsi added a comment to T190090: Offload pings to dedicated server.

Typo above, test IP is 208.80.153.225.
Successfully tested on 1 link with:
cr4-ulsfo> ping source 129.250.204.6 208.80.153.225
Pushing the change to the other transports links, then cr2-codfw.

Mar 20 2019, 9:15 PM · Patch-For-Review, netops, Traffic, Operations
ayounsi added a comment to T187960: Rack/cable/configure asw2-a-eqiad switch stack.

Thanks, opened T218789

Mar 20 2019, 3:17 PM · Wikidata, wikidata-tech-focus, Reading-Infrastructure-Team-Backlog, Cognate, Language-Team, Growth-Team, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi triaged T218789: labsdb1009.mgmt down as High priority.
Mar 20 2019, 3:17 PM · Data-Services, DC-Ops, Operations, ops-eqiad
ayounsi updated the task description for T218751: Audit down ports.
Mar 20 2019, 12:12 AM · DC-Ops, ops-ulsfo, ops-eqiad, Operations
ayounsi added a comment to T218751: Audit down ports.
asw2-d-eqiad
[edit interfaces xe-7/0/4]
+   disable;
Mar 20 2019, 12:11 AM · DC-Ops, ops-ulsfo, ops-eqiad, Operations
ayounsi triaged T218751: Audit down ports as Low priority.
Mar 20 2019, 12:09 AM · DC-Ops, ops-ulsfo, ops-eqiad, Operations

Mar 19 2019

ayounsi closed T213122: Increase network capacity (2018-19 Q3 Goal) as Resolved.
Mar 19 2019, 8:19 PM · Operations, netops
ayounsi edited parent tasks for T187960: Rack/cable/configure asw2-a-eqiad switch stack, added: T218734: Decommission asw-a-eqiad; removed: T208734: Decommission asw-c-eqiad.
Mar 19 2019, 8:19 PM · Wikidata, wikidata-tech-focus, Reading-Infrastructure-Team-Backlog, Cognate, Language-Team, Growth-Team, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi removed a subtask for T208734: Decommission asw-c-eqiad: T187960: Rack/cable/configure asw2-a-eqiad switch stack.
Mar 19 2019, 8:19 PM · decommission, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi added a subtask for T218734: Decommission asw-a-eqiad: T187960: Rack/cable/configure asw2-a-eqiad switch stack.
Mar 19 2019, 8:19 PM · Patch-For-Review, decommission, Operations, ops-eqiad
ayounsi added a parent task for T187960: Rack/cable/configure asw2-a-eqiad switch stack: T208734: Decommission asw-c-eqiad.
Mar 19 2019, 8:18 PM · Wikidata, wikidata-tech-focus, Reading-Infrastructure-Team-Backlog, Cognate, Language-Team, Growth-Team, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi added a subtask for T208734: Decommission asw-c-eqiad: T187960: Rack/cable/configure asw2-a-eqiad switch stack.
Mar 19 2019, 8:18 PM · decommission, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi closed T187960: Rack/cable/configure asw2-a-eqiad switch stack as Resolved.

Everything here is done, thank you all for your help!

Mar 19 2019, 8:17 PM · Wikidata, wikidata-tech-focus, Reading-Infrastructure-Team-Backlog, Cognate, Language-Team, Growth-Team, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi closed T187960: Rack/cable/configure asw2-a-eqiad switch stack, a subtask of T213122: Increase network capacity (2018-19 Q3 Goal), as Resolved.
Mar 19 2019, 8:17 PM · Operations, netops
ayounsi updated the task description for T187960: Rack/cable/configure asw2-a-eqiad switch stack.
Mar 19 2019, 8:17 PM · Wikidata, wikidata-tech-focus, Reading-Infrastructure-Team-Backlog, Cognate, Language-Team, Growth-Team, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi updated the task description for T218734: Decommission asw-a-eqiad.
Mar 19 2019, 8:12 PM · Patch-For-Review, decommission, Operations, ops-eqiad
ayounsi triaged T218734: Decommission asw-a-eqiad as Normal priority.
Mar 19 2019, 8:01 PM · Patch-For-Review, decommission, Operations, ops-eqiad
ayounsi updated the task description for T187960: Rack/cable/configure asw2-a-eqiad switch stack.
Mar 19 2019, 6:45 PM · Wikidata, wikidata-tech-focus, Reading-Infrastructure-Team-Backlog, Cognate, Language-Team, Growth-Team, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi added a comment to T218307: eqiad - eqord Telia link down - IC-314533.

Opened a ticket with Equinix to check the X-connect.

Mar 19 2019, 2:00 PM · Operations, netops

Mar 18 2019

ayounsi added a comment to T218307: eqiad - eqord Telia link down - IC-314533.

Telia did a loop test facing eqiad and our light levels didn't change. While Telia still don't receive light.
The culprit seems to be an active element somewhere on the cr2-eqiad (in DC6)<->Telia (in DC2) X-connect path.

Mar 18 2019, 11:06 PM · Operations, netops

Mar 15 2019

ayounsi claimed T83992: Juniper monitoring.
Mar 15 2019, 7:45 PM · Patch-For-Review, Operations, netops, monitoring
ayounsi triaged T218411: No description on asw2-c-eqiad:xe-2/0/5 as Low priority.
Mar 15 2019, 4:48 PM · Operations, ops-eqiad
ayounsi claimed T218307: eqiad - eqord Telia link down - IC-314533.

We're getting one way light, followed up with Telia.

Mar 15 2019, 4:41 PM · Operations, netops

Mar 14 2019

ayounsi updated the task description for T83992: Juniper monitoring.
Mar 14 2019, 7:44 PM · Patch-For-Review, Operations, netops, monitoring
ayounsi closed T209989: Bird multihop BFD as Resolved.

All done here!

Mar 14 2019, 7:05 PM · Operations, netops
ayounsi added a comment to T209989: Bird multihop BFD.

Followed up on the mailing list:

Mar 14 2019, 7:00 PM · Operations, netops
ayounsi changed the visibility for T218234: Management routers: filter traffic from external to junos-host.
Mar 14 2019, 5:07 PM · Operations, netops
ayounsi closed T218234: Management routers: filter traffic from external to junos-host as Resolved.

All patched. No need for this task to be private anymore.

Mar 14 2019, 5:07 PM · Operations, netops
ayounsi updated the task description for T218234: Management routers: filter traffic from external to junos-host.
Mar 14 2019, 4:45 PM · Operations, netops
ayounsi updated the task description for T218234: Management routers: filter traffic from external to junos-host.
Mar 14 2019, 4:29 PM · Operations, netops
ayounsi added a comment to T190090: Offload pings to dedicated server.

My theory so far, until we can get confirmation from JTAC (as I can't find any doc confirming it or not), is that the firewall action next-ip can only be applied to input filters.
We do know that it works, with input, as it worked when applied to the transit-in4 filter.

Mar 14 2019, 4:15 PM · Patch-For-Review, netops, Traffic, Operations

Mar 13 2019

ayounsi added a comment to T98006: Anycast (Auth)DNS.

Thanks for your comments!

is auth dns already behind LVS.

AuthDNS is not behind LVS and currently have static routes on the routers to redirect the VIPs to the proper machines.
See for example: https://wikitech.wikimedia.org/wiki/Service_restarts#Authoritative_DNS

If possible i would have the dns servers talk BGP directly to the edge routers and have the edge routers configured with ECMP (disclaimer im not failure with LVS so could be unfounded fear)

Indeed. That's what we're currently (slowly) experimenting with, for the recursive DNS servers, see https://wikitech.wikimedia.org/wiki/Anycast_recursive_DNS
If it's successful for the internal recursive DNS, we're considering doing the same with the public Authoritative DNS.

Mar 13 2019, 10:30 PM · Performance-Team (Radar), Patch-For-Review, netops, Operations, Traffic
ayounsi renamed T218234: Management routers: filter traffic from external to junos-host from Management routers: deny traffic from external to junos-host to Management routers: filter traffic from external to junos-host.
Mar 13 2019, 5:26 PM · Operations, netops
ayounsi updated the task description for T218234: Management routers: filter traffic from external to junos-host.
Mar 13 2019, 5:24 PM · Operations, netops
ayounsi triaged T218234: Management routers: filter traffic from external to junos-host as High priority.
Mar 13 2019, 5:20 PM · Operations, netops

Mar 12 2019

ayounsi moved T208734: Decommission asw-c-eqiad from Backlog to pending onsite steps (eqiad) on the decommission board.
Mar 12 2019, 6:30 PM · decommission, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi added a project to T208734: Decommission asw-c-eqiad: decommission.
Mar 12 2019, 6:30 PM · decommission, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi moved T217383: Decommission asw2-a5-eqiad from Backlog to pending onsite steps (eqiad) on the decommission board.
Mar 12 2019, 6:28 PM · decommission, Patch-For-Review, Operations, ops-eqiad
ayounsi added a project to T217383: Decommission asw2-a5-eqiad: decommission.
Mar 12 2019, 6:28 PM · decommission, Patch-For-Review, Operations, ops-eqiad
ayounsi moved T208788: Decommission asw-b-eqiad from Backlog to pending onsite steps (eqiad) on the decommission board.
Mar 12 2019, 6:27 PM · decommission, Operations, ops-eqiad, netops
ayounsi updated subscribers of T217383: Decommission asw2-a5-eqiad.
Mar 12 2019, 6:26 PM · decommission, Patch-For-Review, Operations, ops-eqiad
ayounsi updated subscribers of T208734: Decommission asw-c-eqiad.
Mar 12 2019, 6:26 PM · decommission, Patch-For-Review, Operations, ops-eqiad, netops
ayounsi added a project to T208788: Decommission asw-b-eqiad: decommission.
Mar 12 2019, 6:26 PM · decommission, Operations, ops-eqiad, netops
ayounsi renamed T207706: LibreNMS upgrade to 1.49 from LibreNMS upgrade to 1.44 to LibreNMS upgrade to 1.49.
Mar 12 2019, 5:21 PM · Operations, monitoring
ayounsi added a comment to T209989: Bird multihop BFD.

Victory, the BFD session went up as soon as I changed the TTL to 255.

Mar 12 2019, 3:42 PM · Operations, netops
ayounsi added a comment to T209989: Bird multihop BFD.

Another suggestion from the Bird mailing list.
Junos extensive output mentions Multi-hop min-recv-TTL 254. I'd guess this is set as the router knows that the remote side is in a directly connected network (so assuming a default of 255 - 1 as the session is with the loopback).
Packet capture shows BFD packets from router to server with a TTL or 255, and from server to router with a TTL of 64 (default Linux value).
The theory is that Junos ignores the Bird BFD packets, as 64 < 255.

Mar 12 2019, 3:22 PM · Operations, netops

Mar 11 2019

ayounsi triaged T218059: asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1 flapping as Normal priority.
Mar 11 2019, 9:04 PM · Operations, ops-eqiad