Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (196 w, 2 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Thu, Sep 20

Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

Called HP at 17:48 CDT call ended at 18:25 CDT. went over again the new Log from today and didn't see any internal problem on the server. The only thing he asked me to do it to change the power settings of the server from Dynamic power saving mode to Static High performance mode. The error reported on the integrated management log according (Critical Temperature Threshold Exceeded) to him is an external issue.

Thu, Sep 20, 11:31 PM · ops-codfw, Operations
Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

The server reset again. This time I have something in the log file. see below

Thu, Sep 20, 10:47 PM · ops-codfw, Operations

Wed, Sep 19

Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

@fgiunchedi HW diagnostics didn't report any problem. Upgrade all firmware on the server . we can observe the server all this week and if we do have the same problem again, I will contact HP.

Wed, Sep 19, 5:31 PM · ops-codfw, Operations

Tue, Sep 18

Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

@fgiunchedi getting "The last firmware update attempt was not successful. Ready for the next update." error when trying to update the BIOS. so I email the HP engineer and waiting for response.

Tue, Sep 18, 2:36 PM · ops-codfw, Operations

Mon, Sep 17

Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

Engineer called at 2:20 PM CDT called ended at 2:37 PM CDT. He went over the AHS logs and didn't find any error or issue. His recommendation as I mentioned was to upgrade the BIOS and the controller. @fgiunchedi let me know.

Mon, Sep 17, 8:43 PM · ops-codfw, Operations
Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

@fgiunchedi I open a case with HP and for now They said there is no engineer available to help me, I will be receiving a call back in an hour. Please see below for case information.

Mon, Sep 17, 6:32 PM · ops-codfw, Operations
Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

@fgiunchedi nerver seen this on HP servers. I also checked the log it just says "server reset" no other reason. Since the server is under warranty, I can open a case with HP and submit them the AHS logs they can provide me with more information after they review the log.
If they don't find anything, they will ask me o upgrade the server firmware.

Mon, Sep 17, 6:13 PM · ops-codfw, Operations

Fri, Sep 14

Papaul reassigned T204079: move/setup/install frauth2001.frack.codfw.wmnet from Papaul to Jgreen.

@Jgreen it is all yours. Ping me when you are ready to do the install so you can show me how you do it. Will like to help with install in the future. Thanks.

Fri, Sep 14, 4:09 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations

Thu, Sep 13

Papaul updated the task description for T204079: move/setup/install frauth2001.frack.codfw.wmnet.
Thu, Sep 13, 8:41 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Thu, Sep 13, 5:54 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul updated subscribers of T204079: move/setup/install frauth2001.frack.codfw.wmnet.

I talked with @Jgreen on IRC, he said to use the IP address of frmon2001 for frauth2001 for now since frmon2001 is not installed yet.

Thu, Sep 13, 5:35 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul closed T204173: apply hostname label to cumin2001 / wmf6407 and update visible label field in racktables as Resolved.

Done

Thu, Sep 13, 4:59 PM · Operations, ops-codfw
Papaul closed T204173: apply hostname label to cumin2001 / wmf6407 and update visible label field in racktables, a subtask of T204156: setup/install cumin2001.eqiad.wmnet, as Resolved.
Thu, Sep 13, 4:59 PM · Operations

Wed, Sep 12

Papaul updated subscribers of T204079: move/setup/install frauth2001.frack.codfw.wmnet.

@Jgreen @Dzahn has a comment on https://gerrit.wikimedia.org/r/#/c/operations/dns/+/460127/ the network 10.195.0.72/29 can only hold 6 hosts (2 at the power 3 ) - 2 = 6 . the first usable IP address for that network is 10.195.0.73 and the last usable address is 10.195.0.78 so we can not use 10.195.0.79

Wed, Sep 12, 11:33 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T204079: move/setup/install frauth2001.frack.codfw.wmnet.
Wed, Sep 12, 10:54 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul added a comment to T204079: move/setup/install frauth2001.frack.codfw.wmnet.
papaul@fasw-c-codfw> show interfaces ge-0/0/16 
Physical interface: ge-0/0/16, Enabled, Physical link is Up
Wed, Sep 12, 10:54 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul added a comment to T204079: move/setup/install frauth2001.frack.codfw.wmnet.
papaul@fasw-c-codfw# show | compare 
[edit interfaces interface-range disabled]
-    member ge-0/0/16;
-    member ge-1/0/16;
[edit interfaces interface-range vlan-administration]
     member "ge-[0-1]/0/8" { ... }
+    member "ge-[0-1]/0/16";
[edit interfaces]
+   ge-0/0/16 {
+       description frauth2001:eth0;
+   }
+   ge-1/0/16 {
+       description frauth2001:eth1;
+   }
Wed, Sep 12, 10:50 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T204079: move/setup/install frauth2001.frack.codfw.wmnet.
Wed, Sep 12, 6:10 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul reassigned T194176: wtp2020 correctable memory errors from Papaul to fgiunchedi.
  • Re-seat memory
  • Upgrade BIOS from version 2.3 to 2.6
  • Upgrade IDRAC from version 1.4 to 2.60

Server is back up

Wed, Sep 12, 5:32 PM · Parsing-Team, Operations, ops-codfw
Papaul closed T202700: unrack/decom cr1-eqdfw as Resolved.
Wed, Sep 12, 4:19 PM · Operations, ops-eqdfw
Papaul updated the task description for T202700: unrack/decom cr1-eqdfw.
Wed, Sep 12, 4:19 PM · Operations, ops-eqdfw
Papaul updated the task description for T202700: unrack/decom cr1-eqdfw.
Wed, Sep 12, 4:14 PM · Operations, ops-eqdfw
Papaul updated the task description for T204079: move/setup/install frauth2001.frack.codfw.wmnet.
Wed, Sep 12, 3:32 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations

Mon, Sep 10

Papaul reassigned T202700: unrack/decom cr1-eqdfw from Papaul to ayounsi.
Mon, Sep 10, 3:28 PM · Operations, ops-eqdfw
Papaul updated the task description for T202700: unrack/decom cr1-eqdfw.
Mon, Sep 10, 3:27 PM · Operations, ops-eqdfw
Papaul added a comment to T194176: wtp2020 correctable memory errors.

@fgiunchedi this server is out of warranty. The IDRAC log also is not showing any memory errors and the firmware on the server is way out of date. The first options will be to upgrade the firmware on the server and reboot the severs 2 or 2 times to see .The second option if we stay have the same problem is to pull a memory stick from one of the decommissioned servers.

Mon, Sep 10, 3:02 PM · Parsing-Team, Operations, ops-codfw
Papaul reassigned T203623: Degraded RAID on db2053 from Papaul to Marostegui.

Disk replaced

Mon, Sep 10, 2:45 PM · DBA, Operations, ops-codfw

Thu, Sep 6

Papaul updated the task description for T202700: unrack/decom cr1-eqdfw.
Thu, Sep 6, 3:29 PM · Operations, ops-eqdfw

Wed, Sep 5

This is a test notification, sent at Wed, Sep 5, 9:52 AM.
Wed, Sep 5, 2:52 PM
This is a test notification, sent at Wed, Sep 5, 2:51 PM.
Wed, Sep 5, 2:51 PM

Tue, Sep 4

Papaul added a comment to T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff i will leave it to you so you can play with it tomorrow

Tue, Sep 4, 4:40 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196477: rack/setup/install backup2001.

Here what I get now

Tue, Sep 4, 4:38 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff ok I will change the install in DHCP

Tue, Sep 4, 4:15 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff here is what i get

Tue, Sep 4, 3:46 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T202700: unrack/decom cr1-eqdfw.

@ayounsi I am planning on going to eqdfw this Thursday at 10:30am CDT

Tue, Sep 4, 3:19 PM · Operations, ops-eqdfw
Papaul added a comment to T196477: rack/setup/install backup2001.

sure

Tue, Sep 4, 1:21 PM · Patch-For-Review, ops-codfw, Operations

Fri, Aug 31

Papaul added a comment to T196477: rack/setup/install backup2001.

@Dzahn you can disable those alerts @MoritzMuehlenhoff is running some test on that server.

Fri, Aug 31, 8:10 PM · Patch-For-Review, ops-codfw, Operations

Thu, Aug 30

Papaul added a comment to T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff both the 10GB and 1GB NIC's are already connected to the switch

Thu, Aug 30, 1:15 PM · Patch-For-Review, ops-codfw, Operations

Wed, Aug 29

Papaul updated the task description for T196477: rack/setup/install backup2001.
Wed, Aug 29, 3:42 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196477: rack/setup/install backup2001.

First puppet run complete

Wed, Aug 29, 3:42 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff the installation is complete it is all yours

Wed, Aug 29, 3:17 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff I changed the switch to use the ge-2/0/12 instance of xe-2/0/12 since we are using a 1GB transceiver. the installation is in progress i will let you know when installation is done.

show ethernet-switching table interface ge-2/0/12
Wed, Aug 29, 3:15 PM · Patch-For-Review, ops-codfw, Operations
Papaul placed T202777: add SSDs to wdqs200[12] up for grabs.

@Gehel Disks added to wdqs2001

Wed, Aug 29, 2:39 PM · ops-codfw, Operations, Discovery, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata

Mon, Aug 27

Papaul reassigned T202778: add ssds to wdqs2003 from Papaul to Gehel.

Disks are in place

Mon, Aug 27, 3:34 PM · ops-codfw, Operations, Discovery, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
Papaul reassigned T202822: db2088 rebooted itself and came back sick from Papaul to Marostegui.

BIOS upgrade from version 2.4.3 to 2.8.0
IDRAC upgrade from version 2.40 to 2.60

Mon, Aug 27, 3:28 PM · ops-codfw, Patch-For-Review, Operations, DBA
Papaul reassigned T202824: Degraded RAID on db2058 from Papaul to Marostegui.

Disk replacement complete

Mon, Aug 27, 3:04 PM · DBA, Operations, ops-codfw
Papaul reassigned T201757: Degraded RAID on db2033 from Papaul to Marostegui.

Done

Mon, Aug 27, 2:47 PM · Patch-For-Review, Operations, ops-codfw

Thu, Aug 23

Papaul updated the task description for T196941: Rack/setup cr2-eqdfw.
Thu, Aug 23, 9:22 PM · Patch-For-Review, ops-eqdfw, netops, Operations

Aug 23 2018

Papaul updated the task description for T196941: Rack/setup cr2-eqdfw.
Aug 23 2018, 5:02 PM · Patch-For-Review, ops-eqdfw, netops, Operations
Papaul reassigned T201757: Degraded RAID on db2033 from Papaul to Marostegui.

complete

Aug 23 2018, 2:57 PM · Patch-For-Review, Operations, ops-codfw

Aug 21 2018

Papaul added a comment to T202166: Check/replace PEM2 on cr2-codfw.

Aug 21 2018, 5:04 PM · Operations, ops-codfw
Papaul closed T199530: Rename of wasat to mwmaint2001 (switch labels et al), a subtask of T193915: rename wasat to mwmaint2001 and reinstall it with stretch, as Resolved.
Aug 21 2018, 5:02 PM · Patch-For-Review, Operations
Papaul closed T199530: Rename of wasat to mwmaint2001 (switch labels et al) as Resolved.
Aug 21 2018, 5:02 PM · netops, ops-codfw, Operations
Papaul updated the task description for T199530: Rename of wasat to mwmaint2001 (switch labels et al).
Aug 21 2018, 5:02 PM · netops, ops-codfw, Operations
Papaul reassigned T196417: Rack/Setup frbast2001.frack.codfw.wmnet from Papaul to Jgreen.

@Jgreen Server is back up please confirm and close task.
Thanks.

Aug 21 2018, 4:44 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul closed T202166: Check/replace PEM2 on cr2-codfw as Resolved.
Aug 21 2018, 4:22 PM · Operations, ops-codfw
Papaul added a comment to T202166: Check/replace PEM2 on cr2-codfw.
PEM replaced
Aug 21 2018, 4:21 PM · Operations, ops-codfw

Aug 20 2018

Papaul added a comment to T202166: Check/replace PEM2 on cr2-codfw.

@ayounsi Please see below

Aug 20 2018, 4:37 PM · Operations, ops-codfw
Papaul added a comment to T202166: Check/replace PEM2 on cr2-codfw.

Dear Juniper Networks Customer,

Aug 20 2018, 4:24 PM · Operations, ops-codfw
Papaul added a comment to T202166: Check/replace PEM2 on cr2-codfw.

@ayounsi All lights on all PSU's on cr2 are green. What do you want me to do here?

Aug 20 2018, 3:00 PM · Operations, ops-codfw

Aug 16 2018

Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Aug 16 2018, 3:57 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul reassigned T202006: mw2184 stuck after reboot from Papaul to MoritzMuehlenhoff.

This is what was showing

  • Drain the power
  • Upgrade BIOS from version 2.3.3 to 2.6.0
  • Upgrade IDRAC from 1.4.2 to 2.60

server is back up

Aug 16 2018, 3:37 PM · ops-codfw, Operations

Aug 14 2018

Papaul closed T201440: Decommission labtestnet2001.codfw.wmnet as Resolved.

Complete

Aug 14 2018, 4:41 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations
Papaul updated the task description for T201440: Decommission labtestnet2001.codfw.wmnet.
Aug 14 2018, 4:40 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations
Papaul updated the task description for T201440: Decommission labtestnet2001.codfw.wmnet.
Aug 14 2018, 4:19 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations
Papaul added a comment to T201440: Decommission labtestnet2001.codfw.wmnet.
show | compare    
[edit interfaces interface-range vlan-cloud-hosts1-b-codfw]
-    member ge-5/0/21;
[edit interfaces interface-range cloud-instance-ports]
-    member ge-5/0/32;
[edit interfaces interface-range disabled]
     member ge-6/0/13 { ... }
+    member ge-5/0/21;
+    member ge-5/0/32;
Aug 14 2018, 4:18 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations
Papaul updated the task description for T201440: Decommission labtestnet2001.codfw.wmnet.
Aug 14 2018, 3:29 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations

Aug 13 2018

Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Aug 13 2018, 4:22 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated the task description for T201440: Decommission labtestnet2001.codfw.wmnet.
Aug 13 2018, 3:59 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations
Papaul added a comment to T201440: Decommission labtestnet2001.codfw.wmnet.

Disk wipe in progress

Aug 13 2018, 3:59 PM · Patch-For-Review, ops-codfw, decommission, cloud-services-team, Operations
Papaul added a comment to T196560: rack/setup/install LVS200[7-10].

on boad NICs disable

Aug 13 2018, 3:40 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul reassigned T201761: Degraded RAID on db2039 from Papaul to jcrespo.

Disk replaced

Aug 13 2018, 3:14 PM · Operations, ops-codfw

Aug 9 2018

Papaul closed T187467: Decommission mw2017 as Resolved.
Aug 9 2018, 7:34 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul updated the task description for T187467: Decommission mw2017.
Aug 9 2018, 5:13 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul added a comment to T187467: Decommission mw2017.

switch port information
ge-3/0/16

show interfaces ge-3/0/16 
Physical interface: ge-3/0/16, Administratively down, Physical link is Down
  Interface index: 1187, SNMP ifIndex: 557
  Description: DISABLED
Aug 9 2018, 5:12 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul updated the task description for T187467: Decommission mw2017.
Aug 9 2018, 4:54 PM · Patch-For-Review, decommission, ops-codfw, Operations

Aug 8 2018

Papaul closed T195228: db2064 crashed and totally broken - decommission it as Resolved.

This is complete resolving it.

Aug 8 2018, 4:08 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T195228: db2064 crashed and totally broken - decommission it.
Aug 8 2018, 4:07 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T195228: db2064 crashed and totally broken - decommission it.
Aug 8 2018, 3:47 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T195228: db2064 crashed and totally broken - decommission it.
 show interfaces ge-6/0/12 
Physical interface: ge-6/0/12, Administratively down, Physical link is Down
  Interface index: 1212, SNMP ifIndex: 761
  Description: DISABLED
Aug 8 2018, 3:47 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T187467: Decommission mw2017.
Aug 8 2018, 3:26 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul added a comment to T187467: Decommission mw2017.

Disk wipe in progress

Aug 8 2018, 3:26 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul updated the task description for T195228: db2064 crashed and totally broken - decommission it.
Aug 8 2018, 3:24 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T196941: Rack/setup cr2-eqdfw.

@ayounsi all SFP+-10G-LR are in place .

Aug 8 2018, 2:57 PM · Patch-For-Review, ops-eqdfw, netops, Operations
Papaul closed T201387: Upgrade pc2004 and pc2005 BIOS as Resolved.

@Marostegui complete closing the task. Thanks

Aug 8 2018, 1:49 PM · Patch-For-Review, Operations, ops-codfw, DBA
Papaul updated the task description for T201387: Upgrade pc2004 and pc2005 BIOS.
Aug 8 2018, 1:49 PM · Patch-For-Review, Operations, ops-codfw, DBA
Papaul updated the task description for T195228: db2064 crashed and totally broken - decommission it.
Aug 8 2018, 1:39 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA

Aug 6 2018

Papaul closed Unknown Object (Task), a subtask of T196941: Rack/setup cr2-eqdfw, as Resolved.
Aug 6 2018, 9:44 PM · Patch-For-Review, ops-eqdfw, netops, Operations
Papaul added a comment to T199821: connect eth1 on labtestnet2002 and labtestnet2003.

switch port is enable and link is up

show interfaces ge-1/0/17 descriptions 
Interface       Admin Link Description
ge-1/0/17       up    up   labtestnet2003:eth1
papaul@asw-b-codfw# show interfaces interface-range cloud-instance-ports 
member ge-5/0/32;
member ge-5/0/31;
member ge-5/0/30;
member ge-1/0/16;
member ge-1/0/17;
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members cloud-instances1-b-codfw;
        }
    }
}
Aug 6 2018, 4:26 PM · Operations, netops, ops-codfw, cloud-services-team
Papaul added a comment to T195228: db2064 crashed and totally broken - decommission it.

switch port information

Aug 6 2018, 4:00 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T195228: db2064 crashed and totally broken - decommission it.

Moved all disks in one of the decom server (db2013). Disk wipe in progress

Aug 6 2018, 4:00 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T201245: Degraded RAID on db2054.

yes thanks

Aug 6 2018, 3:59 PM · DBA, Operations, ops-codfw
Papaul added a comment to T201245: Degraded RAID on db2054.
Aug 6 2018, 3:57 PM · DBA, Operations, ops-codfw
Papaul added a comment to T201245: Degraded RAID on db2054.

Done

Aug 6 2018, 3:38 PM · DBA, Operations, ops-codfw
Papaul reassigned T200641: pc2006 rebooted itself from Papaul to Marostegui.

@Marostegui Bios update to 2.8.0 . It is all yours

Aug 6 2018, 3:36 PM · Operations, ops-codfw, DBA
Papaul added a comment to T200641: pc2006 rebooted itself.

This server has pro support as mentioned in T139283

Aug 6 2018, 3:14 PM · Operations, ops-codfw, DBA
Papaul reassigned T201245: Degraded RAID on db2054 from Papaul to Marostegui.

@Marostegui Disk replaced

Aug 6 2018, 2:50 PM · DBA, Operations, ops-codfw
Papaul triaged T201245: Degraded RAID on db2054 as Normal priority.
Aug 6 2018, 1:46 PM · DBA, Operations, ops-codfw

Aug 2 2018

Papaul reassigned T199821: connect eth1 on labtestnet2002 and labtestnet2003 from Papaul to Andrew.

@Andrew both ports are up and in the cloud-instance-ports interfaces ranges. Please check if everything looks good, you can resolve the task.

Aug 2 2018, 5:02 PM · Operations, netops, ops-codfw, cloud-services-team
Papaul added a comment to T199821: connect eth1 on labtestnet2002 and labtestnet2003.

Port information
labtestnet2002 rack B1 ge-1/0/16
labtestnet2003 rack B1 ge1/0/17

Aug 2 2018, 4:54 PM · Operations, netops, ops-codfw, cloud-services-team