Page MenuHomePhabricator

ayounsi (Arzhel Younsi)
Network Engineer

Projects (6)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Apr 3 2017, 6:23 PM (116 w, 1 d)
Availability
Available
IRC Nick
xionox
LDAP User
Ayounsi
MediaWiki User
AYounsi (WMF) [ Global Accounts ]

Recent Activity

Today

ayounsi renamed T226331: Upgrade Netbox to 2.6.1 from Upgrade Netbox to 2.6.0 to Upgrade Netbox to 2.6.1.
Tue, Jun 25, 3:35 PM · netbox
ayounsi added a comment to T226331: Upgrade Netbox to 2.6.1.

2.6.1 is out: https://github.com/digitalocean/netbox/releases/tag/v2.6.1
Mostly bug fixes.

Tue, Jun 25, 3:31 PM · netbox
ayounsi added a comment to T220669: RPKI Validation.

New proposal after talking to @faidon. The one above focuses on dropping invalids. The one bellow adds visibility on Invalid (until we drop them) as well as unknown and valids.

Tue, Jun 25, 3:23 PM · Operations, netops
ayounsi added a comment to T219486: Send peering requests to AS with the worst TTFB.

Current process is to lookup their email contact in PeeringDB and manually send them an email, CCing our peering@ email.

Tue, Jun 25, 3:01 PM · Traffic, Operations, Performance-Team
ayounsi updated the task description for T226424: update RE-S-X6-64G-S in cr[12]-eqiad.
Tue, Jun 25, 8:14 AM · Operations, netops, ops-eqiad
ayounsi updated the task description for T226422: update RE-S-X6-64G-S in cr[12]-codfw.
Tue, Jun 25, 7:04 AM · netops, ops-codfw, Operations
ayounsi closed T226394: Telia IC-307235 reported down from the eqiad side as Resolved.

Tha faulty card replaced and at 2019-06-25 05:41 UTC the circuit recovered and running at the moment , please check and let us know if you have any issue .
Thanks for your patience and understanding

Tue, Jun 25, 6:10 AM · Operations, netops
ayounsi added a comment to T226444: rack/setup/install ganeti400[123].

asw-ulsfo.mgmt.ulsfo.wmnet was the old stack, asw2-ulsfo.mgmt.ulsfo.wmnet is the way to go.

Tue, Jun 25, 5:46 AM · Operations, ops-ulsfo

Yesterday

ayounsi closed T226405: Remove access to network gear for Casey Dentinger as Resolved.

User (set as read-only user) removed from all network devices.

Mon, Jun 24, 5:01 PM · Operations, netops
ayounsi claimed T226394: Telia IC-307235 reported down from the eqiad side.
Mon, Jun 24, 3:32 PM · Operations, netops
ayounsi added a comment to T226407: Replace cr[1-2].codfw fan filters .

Feel free to do it anytime.
Doc is on https://www.juniper.net/documentation/en_US/release-independent/junos/topics/topic-map/mx480-maintain-cooling-system.html

Mon, Jun 24, 3:31 PM · netops, ops-codfw, Operations, DC-Ops
ayounsi added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

That sounds reasonable for the PoC, depending on rack space. @faidon for the last word.

Mon, Jun 24, 11:53 AM · ops-eqiad, Operations, cloud-services-team (Kanban), Cloud-Services
ayounsi added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

our network interface saturation monitoring is still diamond-based

What does that mean?

Mon, Jun 24, 6:12 AM · ops-eqiad, Operations, cloud-services-team (Kanban), Cloud-Services

Sun, Jun 23

ayounsi triaged T226331: Upgrade Netbox to 2.6.1 as Low priority.
Sun, Jun 23, 1:06 PM · netbox

Thu, Jun 20

herron awarded T224128: Migrate network device syslogs to Kafka logging pipeline a Party Time token.
Thu, Jun 20, 1:55 PM · Patch-For-Review, User-herron, Operations, netops, Wikimedia-Logstash
ayounsi closed T225905: check_ospf.py fails on mr1-eqsin as Resolved.

Check is now green!

Thu, Jun 20, 8:33 AM · Patch-For-Review, Operations, netops
ayounsi closed T224535: Investigate cr2-eqord's disconnection from the rest of the network as Resolved.

Opened T226158 for the tunnel. Everything else here is done.

Thu, Jun 20, 8:11 AM · Operations, netops
ayounsi added a comment to T225905: check_ospf.py fails on mr1-eqsin.
(timeout is not respected)

I couldn't reproduce that bug. And it seems to be respected on all my tries now.
CR above should solve the issue.

Thu, Jun 20, 6:58 AM · Patch-For-Review, Operations, netops

Wed, Jun 19

ayounsi added a comment to T211728: Outbound BGP graceful shutdown.

Interesting! This would be useful before doing maintenance on a whole router.
I opened an issue upstream asking for a per AS option, see https://github.com/mwiget/bgp_graceful_shutdown/issues/1

Wed, Jun 19, 4:34 PM · Operations, netops
ayounsi closed T224128: Migrate network device syslogs to Kafka logging pipeline as Resolved.

All done here!

Wed, Jun 19, 3:01 PM · Patch-For-Review, User-herron, Operations, netops, Wikimedia-Logstash

Tue, Jun 18

ayounsi closed T222967: Interface errors on cr1-codfw: xe-5/3/1 as Resolved.

The issue should be resolved as per our second line , and any further optimization on the circuit will be done through maintenance window and of course you will be notified, so please check and let us know if you still see any issue .

Tue, Jun 18, 12:29 PM · Operations, ops-codfw
ayounsi added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

There is no security or technical limitation on having 1 interface in each vlan as long as they don't go across rows. Hosts in one vlan can already reach the hosts in the other.
So the question is more about using extra 10G switch ports, increasing configuration complexity, and load (for the cluster in general).

Tue, Jun 18, 5:55 AM · ops-eqiad, Operations, cloud-services-team (Kanban), Cloud-Services
ayounsi added a comment to T225704: eqiad: rack/setup/install (4) dbproxy systems..

No differences other than the physical row they're in. They will be able to reach the same resources.

Tue, Jun 18, 5:27 AM · Patch-For-Review, Operations, DBA
ayounsi added a comment to T225704: eqiad: rack/setup/install (4) dbproxy systems..

dbproxy1001-1008 are in the private vlans across row A-B-C, none in D. Is row D private fine for dbproxy1020/1021 or should they be in private-A/B/C ?

Tue, Jun 18, 5:22 AM · Patch-For-Review, Operations, DBA
ayounsi added a comment to T225704: eqiad: rack/setup/install (4) dbproxy systems..

Ok good so that's 1018/1019).
In which vlan the other 2 (dbproxy1020/1021) need to go then?

Tue, Jun 18, 5:16 AM · Patch-For-Review, Operations, DBA
ayounsi added a comment to T225704: eqiad: rack/setup/install (4) dbproxy systems..

@Cmjohnson I'm pretty sure they will have to move :( Creating a new vlan and all the supporting config (DHCP, routing, IP allocation, etc) is a non trivial task for a vlan that is not expected to grow past those 2 hosts.

Tue, Jun 18, 5:09 AM · Patch-For-Review, Operations, DBA

Mon, Jun 17

ayounsi reassigned T225704: eqiad: rack/setup/install (4) dbproxy systems. from ayounsi to Cmjohnson.

The cloud support vlan/network is legacy, so I'd rather not create a new one (in a new row).
As we already have cloud-support1-a-eqiad and cloud-support1-c-eqiad (row A and C) could dbproxy1020/1021 be in row A instead?

Mon, Jun 17, 6:18 PM · Patch-For-Review, Operations, DBA
ayounsi closed T224180: Send some LibreNMS alerts to dcops and netops only as Resolved.

I made the change to email dcops + me for those alerts.

Mon, Jun 17, 8:32 AM · Operations, DC-Ops, netops, observability
ayounsi triaged T225905: check_ospf.py fails on mr1-eqsin as Normal priority.
Mon, Jun 17, 7:51 AM · Patch-For-Review, Operations, netops
ayounsi triaged T225904: Mjolnir bulk update failure check - eqiad as High priority.
Mon, Jun 17, 6:54 AM · Discovery-Search
ayounsi raised the priority of T222651: Tracking failures in your Matomo Analytics from Low to High.

Increasing priority as they are regularly being sent to the whole SRE team.

Mon, Jun 17, 4:29 AM · Analytics

Thu, Jun 13

ayounsi added a comment to T225720: poll power data for redeployment of esams/knams.

I don't think Junos have that feature.

Thu, Jun 13, 1:17 PM · Traffic, DC-Ops, Operations
ayounsi added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

@ayounsi I am planning on installing those new servers in row c and row D and I don't have the "interface-range ganeti" in both of those rows Is it okay for me to go ahead and create "interface-range ganeti" on asw-c-codfw and asw-d-codfw?

Thu, Jun 13, 7:55 AM · Patch-For-Review, ops-codfw, Operations

Wed, Jun 12

ayounsi triaged T225598: webpagetest-alerts grafana alert as High priority.
Wed, Jun 12, 9:17 AM · Performance-Team
ayounsi reopened T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): as "Open".

DISK WARNING - free space: / 3588 MB (8% inode=63%):

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=contint1001&service=Disk+space#comments

Wed, Jun 12, 9:05 AM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure, Operations
ayounsi reopened T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%):, a subtask of T207707: contint1001 store docker images on separate partition or disk, as Open.
Wed, Jun 12, 9:05 AM · serviceops, Release-Engineering-Team (Kanban), Operations, Continuous-Integration-Infrastructure

Fri, Jun 7

ayounsi triaged T225296: High Prometheus TCP retransmits as Normal priority.
Fri, Jun 7, 11:00 AM · User-fgiunchedi, Cloud-Services, observability, Analytics
ayounsi triaged T225284: thumbor haproxy trying to send syslog on wrong port as High priority.
Fri, Jun 7, 9:30 AM · User-jijiki, serviceops, observability, Thumbor
ayounsi added a comment to T225137: codfw humidity too high.

Some PSUs humidity alerts were flapping so I disabled them.
https://librenms.wikimedia.org/device/device=107/tab=edit/section=health/
https://librenms.wikimedia.org/device/device=108/tab=edit/section=health/
To be re-enabled when humidity levels are back to normal.

Fri, Jun 7, 9:08 AM · Operations, ops-codfw
ayounsi triaged T225280: stat1007 disk space warning as High priority.
Fri, Jun 7, 8:23 AM · Analytics-Kanban, Analytics
ayounsi added a project to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN.: ops-eqiad.

Great, over to @Cmjohnson then!

Fri, Jun 7, 8:04 AM · ops-eqiad, Operations, netops, Analytics

Thu, Jun 6

ayounsi added a comment to T224682: notebook1003:/srv/ 2% disk space left.

Back to warning:

DISK WARNING - free space: /srv 6010 MB (4% inode=83%):

Thu, Jun 6, 12:16 PM · User-Elukey, Operations, Analytics

Wed, Jun 5

ayounsi added a comment to T223458: mgmt outages for cloud* systems seem to page everyone.

On the designate-sink issue:

Wed, Jun 5, 9:03 PM · cloud-services-team (Kanban)
ayounsi triaged T225140: Icinga alerts that should open tasks instead of alerting as Low priority.
Wed, Jun 5, 7:24 PM · observability
ayounsi reopened T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): as "Open".

Alert is warning again.

DISK WARNING - free space: / 4594 MB (10% inode=57%):

Wed, Jun 5, 7:21 PM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure, Operations
ayounsi reopened T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%):, a subtask of T207707: contint1001 store docker images on separate partition or disk, as Open.
Wed, Jun 5, 7:21 PM · serviceops, Release-Engineering-Team (Kanban), Operations, Continuous-Integration-Infrastructure
ayounsi added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

What are the needed network changes?

Wed, Jun 5, 5:55 PM · ops-eqiad, Operations, netops, Analytics
ayounsi claimed T225108: Prometheus logs showing errors for routinator .
Wed, Jun 5, 4:04 PM · observability, netops, Operations
ayounsi added a comment to T225108: Prometheus logs showing errors for routinator .

Opened https://github.com/NLnetLabs/routinator/issues/154 upstream.

Wed, Jun 5, 4:04 PM · observability, netops, Operations
ayounsi added a parent task for T225108: Prometheus logs showing errors for routinator : T220669: RPKI Validation.
Wed, Jun 5, 3:45 PM · observability, netops, Operations
ayounsi added a subtask for T220669: RPKI Validation: T225108: Prometheus logs showing errors for routinator .
Wed, Jun 5, 3:45 PM · Operations, netops
ayounsi updated subscribers of T225108: Prometheus logs showing errors for routinator .

Data collection stopped after the upgrade to Routinator 0.4.0:
https://grafana.wikimedia.org/d/UwUa77GZk/rpki?refresh=5m&orgId=1&from=now-7d&to=now

Wed, Jun 5, 3:38 PM · observability, netops, Operations

Tue, Jun 4

ayounsi closed T220194: Degraded RAID on cp3034 as Invalid.

This seems to be due to a transient connectivity issue, and not a RAID issue.

Tue, Jun 4, 10:29 PM · ops-esams, Operations
ayounsi closed T220193: Degraded RAID on cp3041 as Invalid.

This seems to be due to a transient connectivity issue, and not a RAID issue.

Tue, Jun 4, 10:29 PM · ops-esams, Operations
ayounsi triaged T225035: cp3035 PS Redundancy Lost as High priority.
Tue, Jun 4, 10:23 PM · Traffic, ops-esams, Operations
ayounsi claimed T224128: Migrate network device syslogs to Kafka logging pipeline.

Test was successful, next step is to do the change to all devices.

Tue, Jun 4, 9:36 PM · Patch-For-Review, User-herron, Operations, netops, Wikimedia-Logstash
ayounsi added a comment to T224535: Investigate cr2-eqord's disconnection from the rest of the network.

1/ outstanding alert, I think this is due to the alert being triggered right before a 3 days weekend and people not paying enough attention to active Icinga alerts. This should be tackled at a SRE wide scope.

Tue, Jun 4, 4:25 PM · Operations, netops
ayounsi updated subscribers of T220669: RPKI Validation.

Bellow is my proposal to add validation to our config. There are many possible ways of doing it, so feedback from @faidon or @mark is welcome!

Tue, Jun 4, 12:16 AM · Operations, netops

Mon, Jun 3

ayounsi closed T224627: Requesting access to ops group in admin for jeh as Resolved.

You should be good, please reopen if any issues.

Mon, Jun 3, 6:28 PM · Operations, SRE-Access-Requests
ayounsi closed T224627: Requesting access to ops group in admin for jeh, a subtask of T224192: Onboard jhedden to Wikimedia Foundation as SRE in Cloud Services, as Resolved.
Mon, Jun 3, 6:28 PM · Patch-For-Review, cloud-services-team (Kanban)
ayounsi added a comment to T221507: Netbox report to validate network equipment data.

If you need any examples, that's what I do in: https://gerrit.wikimedia.org/r/c/operations/software/netbox-deploy/+/507217/3/scripts/j2nb.py#46 (virtual chassis detection)

Mon, Jun 3, 3:19 PM · Patch-For-Review, netbox, User-crusnov, Operations-Software-Development, Operations, netops
ayounsi added a comment to T224888: Network port utilization alerts should be paging .

I agree!

Mon, Jun 3, 3:00 PM · Traffic, Operations, netops

Fri, May 31

ayounsi added a comment to T221507: Netbox report to validate network equipment data.
Fri, May 31, 8:36 PM · Patch-For-Review, netbox, User-crusnov, Operations-Software-Development, Operations, netops
ayounsi closed T222788: Request to be added to the ldap/wmde group as Resolved.

Everything should be set, please reopen the task if any issue.

Fri, May 31, 6:25 PM · Patch-For-Review, WMF-Legal, LDAP-Access-Requests, Operations, WMF-NDA-Requests
ayounsi closed T224597: Add Jhedden to ops,wmf LDAP groups as Resolved.

Not sure who did it but the uid jeh is already in both wmf and ops LDAP groups.

Fri, May 31, 6:18 PM · LDAP-Access-Requests
ayounsi closed T224597: Add Jhedden to ops,wmf LDAP groups, a subtask of T224192: Onboard jhedden to Wikimedia Foundation as SRE in Cloud Services, as Resolved.
Fri, May 31, 6:18 PM · Patch-For-Review, cloud-services-team (Kanban)
ayounsi closed T224507: Remove user Greta WMDE from wmde LDAP group as Resolved.

User greta removed from the wmde LDAP group.

Fri, May 31, 4:57 PM · LDAP-Access-Requests, Operations

Thu, May 30

ayounsi triaged T224706: Debian mirror in sync with upstream as High priority.
Thu, May 30, 11:25 PM · Operations
ayounsi updated the task description for T224692: mx1001 exim queue warning.
Thu, May 30, 7:13 PM · Operations, observability
ayounsi triaged T224694: cp3041 - Varnish frontend child restarted icinga alert as Normal priority.
Thu, May 30, 7:12 PM · Patch-For-Review, Operations, Traffic
ayounsi updated the task description for T224692: mx1001 exim queue warning.
Thu, May 30, 7:05 PM · Operations, observability
ayounsi triaged T224692: mx1001 exim queue warning as High priority.
Thu, May 30, 7:02 PM · Operations, observability
ayounsi lowered the priority of T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): from Unbreak Now! to High.
Thu, May 30, 6:53 PM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure, Operations
ayounsi reopened T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): as "Open".

Reopening as it alerted again: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=contint1001&service=Disk+space

Thu, May 30, 6:53 PM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure, Operations
ayounsi reopened T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%):, a subtask of T207707: contint1001 store docker images on separate partition or disk, as Open.
Thu, May 30, 6:53 PM · serviceops, Release-Engineering-Team (Kanban), Operations, Continuous-Integration-Infrastructure
ayounsi assigned T213417: lvs2002: raid battery failure to Papaul.

Assigning that to Papaul as I think next steps are for DCops, but please reassign if incorrect.

Thu, May 30, 6:42 PM · Operations, ops-codfw, Traffic
ayounsi updated the task description for T224688: Outstanding icinga critical on cloudcontrol-dev hosts.
Thu, May 30, 6:36 PM · cloud-services-team (Kanban), Cloud-Services
ayounsi triaged T224688: Outstanding icinga critical on cloudcontrol-dev hosts as Low priority.
Thu, May 30, 6:35 PM · cloud-services-team (Kanban), Cloud-Services
ayounsi updated the task description for T224682: notebook1003:/srv/ 2% disk space left.
Thu, May 30, 5:33 PM · User-Elukey, Operations, Analytics
ayounsi triaged T224682: notebook1003:/srv/ 2% disk space left as High priority.
Thu, May 30, 5:33 PM · User-Elukey, Operations, Analytics
ayounsi added a comment to T221507: Netbox report to validate network equipment data.

Opened T224679

Thu, May 30, 4:48 PM · Patch-For-Review, netbox, User-crusnov, Operations-Software-Development, Operations, netops

Wed, May 29

ayounsi closed T224511: cr1-codfw linecard failure as Resolved.

Everything seems back to normal. Please reopen if the same issue happen again and we will proceed with a RMA.

Wed, May 29, 7:47 PM · Operations, netops
ayounsi updated the task description for T224511: cr1-codfw linecard failure.
Wed, May 29, 7:47 PM · Operations, netops
ayounsi added a comment to T221507: Netbox report to validate network equipment data.
  • The cr1-eqsin serial change is a bit odd. Netbox used to have a record of what Juniper reports as the "midplane" serial number, not the "chassis". This was changed, but the midplane was what we had from the invoice as well -- so note that the Accounting report is now error'ing out instead.
  • asw-N-eqiad serial changes above -- these are now inconsistent with what we have from the Accounting side (so the report fails now). This needs further investigation for which one is ground truth?
Wed, May 29, 7:32 PM · Patch-For-Review, netbox, User-crusnov, Operations-Software-Development, Operations, netops
ayounsi added a comment to T220669: RPKI Validation.

Next step is to configure the RPKI validators on one router (eg. cr4-ulsfo):

[edit routing-options]
+   validation {
+       group rpki {
+           session 10.64.32.19 {
+               port 3323;
+           }
+           session 10.192.0.103 {
+               port 3323;
+           }
+       }
+   }

Note that this will not do any routing changes.

Wed, May 29, 7:11 PM · Operations, netops

Tue, May 28

ayounsi added a comment to T220639: Show IPs matching a list of IP subnets in Webrequest data.

the only missing thing is the long list of RPKI invalid prefixes

Tue, May 28, 8:34 PM · User-Elukey, Analytics
ayounsi claimed T224477: rhenium [spare] server still receiving flow data.

Network devices have to have their target changed, see T212011.
Note that only cr2-eqiad is actively sending netflow, the other routers are only sending keepalives.

Tue, May 28, 7:39 PM · Traffic, Operations
ayounsi added a comment to T212011: migrate netinsights from rhenium to sulfur.

task description says sulfur.wikimedia.org but the link is to sodium ( https://netbox.wikimedia.org/dcim/devices/1171/ )
Which one is correct?

Tue, May 28, 7:26 PM · netops, Operations
ayounsi added a subtask for T212011: migrate netinsights from rhenium to sulfur: T224477: rhenium [spare] server still receiving flow data.
Tue, May 28, 7:25 PM · netops, Operations
ayounsi added a parent task for T224477: rhenium [spare] server still receiving flow data: T212011: migrate netinsights from rhenium to sulfur.
Tue, May 28, 7:25 PM · Traffic, Operations
ayounsi added a comment to T221507: Netbox report to validate network equipment data.

Note that this already helped find inconsistencies:

Tue, May 28, 7:21 PM · Patch-For-Review, netbox, User-crusnov, Operations-Software-Development, Operations, netops
ayounsi added a comment to T224511: cr1-codfw linecard failure.

I picked 2h for the sake of picking a number that sounds right, but it's not backed by anything. Any value works for me.

Tue, May 28, 6:32 PM · netops, Operations
ayounsi updated the task description for T224511: cr1-codfw linecard failure.
Tue, May 28, 6:05 PM · netops, Operations
ayounsi triaged T224511: cr1-codfw linecard failure as High priority.
Tue, May 28, 6:03 PM · netops, Operations

May 24 2019

ayounsi updated the task description for T220669: RPKI Validation.
May 24 2019, 1:00 AM · Operations, netops
ayounsi reassigned T224223: decommission lvs100[123456].wikimedia.org from ayounsi to RobH.
May 24 2019, 12:43 AM · Traffic, Operations, ops-eqiad, DC-Ops, decommission
ayounsi added a comment to T224223: decommission lvs100[123456].wikimedia.org.
cr1/2-eqiad
[edit firewall family inet filter border-in4]
-      /* workaround until lvs1001-lvs1007 are decom'ed */
-      term protect-old-lvs-servers {
-          from {
-              destination-address {
-                  208.80.154.55/32;
-                  208.80.154.56/32;
-                  208.80.154.57/32;
-                  208.80.154.137/32;
-                  208.80.154.138/32;
-                  208.80.154.139/32;
-              }
-              protocol tcp;
-              destination-port [ 22 179 9090 9100 ];
-          }
-          then {
-              discard;
-          }
-      }
May 24 2019, 12:27 AM · Traffic, Operations, ops-eqiad, DC-Ops, decommission

May 23 2019

ayounsi triaged T224250: Setup new msw1-codfw as Normal priority.
May 23 2019, 9:23 PM · ops-codfw, Operations, netops
ayounsi updated the task description for T224223: decommission lvs100[123456].wikimedia.org.
May 23 2019, 6:06 PM · Traffic, Operations, ops-eqiad, DC-Ops, decommission

May 22 2019

ayounsi added a comment to T224180: Send some LibreNMS alerts to dcops and netops only.

I'd also prefer it, but implementing it is not a trivial task.
Emails to DCops would still be a step forward compared to the current situation where most of SRE is emailed (noc@) and I'm the one opening the tasks :)

May 22 2019, 11:14 PM · Operations, DC-Ops, netops, observability