Page MenuHomePhabricator
Feed Advanced Search

Tue, Nov 5

faidon triaged T237466: Remove unused custom fields from Netbox as Normal priority.
Tue, Nov 5, 9:20 PM · SRE-tools, DC-Ops, netbox
faidon added a comment to T233318: scs monitoring missing in Icinga.

Thanks @herron! Should we resolve this?

Tue, Nov 5, 3:16 PM · Icinga, observability, Operations

Mon, Nov 4

faidon changed the status of T204589: eqiad: (1) misc single cpu server allocation for performance browser testing from Open to Stalled.

Update per IRC conversation with @Gilles: this is still needed, but is stalled and currently blocked until the Performance-Team fills its vacancy.

Mon, Nov 4, 10:23 AM · Performance-Team (Radar), Operations, hardware-requests

Fri, Nov 1

faidon added a parent task for T235669: codfw: recycle Cisco old servers: T128821: reclaim and return all cisco servers.
Fri, Nov 1, 2:59 PM · Operations, ops-codfw
faidon added a subtask for T128821: reclaim and return all cisco servers: T235669: codfw: recycle Cisco old servers.
Fri, Nov 1, 2:59 PM · decommission, Goal, Operations
faidon renamed T235669: codfw: recycle Cisco old servers from Recycle Cisco old servers to codfw: recycle Cisco old servers.
Fri, Nov 1, 2:58 PM · Operations, ops-codfw
faidon added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T237030: setup new MX204 in knams.
Fri, Nov 1, 2:53 PM · Patch-For-Review, DC-Ops, ops-esams, Operations
faidon added a parent task for T237030: setup new MX204 in knams: T235805: ESAMS Refresh/Rebuild (October 2019).
Fri, Nov 1, 2:53 PM · netops, ops-esams, Operations
faidon closed T237011: Update DNS/NTP servers on the esams PDUs/SCS as Resolved.

Anycasting NTP sounds a good idea in general, but a) should be kept in a separate task b) it doesn't sound like a priority IMHO at this time. Things work OK, and that sounds like a time investment that won't pay off right now.

Fri, Nov 1, 2:51 PM · DC-Ops, ops-esams, Operations
faidon reassigned T214024: Two test hosts for SREs from faidon to RobH.

Ping :)

Fri, Nov 1, 2:37 PM · Operations, hardware-requests

Thu, Oct 31

faidon added a comment to T233774: Netbox: tracking of hardware errors / grouping servers in order/batches.

Indeed, and in fact procurement task alone would be enough to identify the batch. Is that what you were looking for @MoritzMuehlenhoff? How could we make this more visible?

Thu, Oct 31, 5:03 PM · Operations, netbox
faidon moved T236767: cr3-esams:et-1/0/0 flap from Backlog to Next visit on the ops-esams board.
Thu, Oct 31, 3:06 PM · ops-esams, Operations
faidon moved T237006: Relabel cables with duplicate IDs from Backlog to Next visit on the ops-esams board.
Thu, Oct 31, 3:06 PM · ops-esams, Operations
faidon moved T237009: Add missing labels for equipment and cables from Backlog to Next visit on the ops-esams board.
Thu, Oct 31, 3:06 PM · DC-Ops, ops-esams, Operations
faidon moved T237014: Update spare QFX labels from Backlog to Next visit on the ops-esams board.
Thu, Oct 31, 3:06 PM · ops-esams, Operations
faidon updated subscribers of T236216: rack/setup/install ganeti300[123].

@BBlack, what were your plans here? Can others in SRE help with some of that perhaps?

Thu, Oct 31, 2:04 PM · Operations, ops-esams
faidon renamed T237011: Update DNS/NTP servers on the esams PDUs/SCS from Update DNS/NTP for all non-network/server gear to Update DNS/NTP servers on the esams PDUs/SCS.
Thu, Oct 31, 1:51 PM · DC-Ops, ops-esams, Operations
faidon triaged T237011: Update DNS/NTP servers on the esams PDUs/SCS as Normal priority.
Thu, Oct 31, 1:45 PM · DC-Ops, ops-esams, Operations
faidon added a comment to T184066: rack/setup/install ps[12]-oe1[456]-esams.

@RobH what's the status of this?

Thu, Oct 31, 1:41 PM · Operations, ops-esams
faidon triaged T237009: Add missing labels for equipment and cables as Normal priority.
Thu, Oct 31, 1:36 PM · DC-Ops, ops-esams, Operations
faidon renamed T237007: Add a Netbox check for duplicate cable IDs from Add a Netbox coherence check for duplicate cable IDs to Add a Netbox check for duplicate cable IDs.
Thu, Oct 31, 1:15 PM · Patch-For-Review, DC-Ops, SRE-tools, netbox
faidon closed T184061: SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking), a subtask of T235805: ESAMS Refresh/Rebuild (October 2019), as Resolved.
Thu, Oct 31, 1:07 PM · Patch-For-Review, DC-Ops, ops-esams, Operations
faidon closed T184061: SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking) as Resolved.
Thu, Oct 31, 1:07 PM · Operations, Epic, ops-esams
faidon created T237007: Add a Netbox check for duplicate cable IDs.
Thu, Oct 31, 1:05 PM · Patch-For-Review, DC-Ops, SRE-tools, netbox
faidon triaged T237006: Relabel cables with duplicate IDs as Normal priority.
Thu, Oct 31, 1:01 PM · ops-esams, Operations
faidon moved T185337: rack spare switches in c1-eqiad from Backlog to Watching on the netops board.
Thu, Oct 31, 1:02 AM · Operations, netops, ops-eqiad
faidon moved T208734: Decommission asw-c-eqiad from Backlog to Watching on the netops board.
Thu, Oct 31, 1:01 AM · decommission, Operations, ops-eqiad, netops
faidon moved T208788: Decommission asw-b-eqiad from Backlog to Watching on the netops board.
Thu, Oct 31, 1:01 AM · decommission, Operations, ops-eqiad, netops
faidon closed T82038: create a test for multicast relay as Declined.

We've lived since 2013 without it, I don't think the extra complexity here would be warranted!

Thu, Oct 31, 1:00 AM · observability, netops, Operations
faidon closed T124843: Peer with SFMIX at ULSFO in 200 Paul as Declined.

We reached out a while ago and based on the conversations there, I don't think it's good idea right now.

Thu, Oct 31, 12:59 AM · netops, Operations
faidon placed T211728: Outbound BGP graceful shutdown up for grabs.

Reviving this... is this still waiting for my feedback? If so, LGTM :)

Thu, Oct 31, 12:56 AM · Operations, netops

Wed, Oct 30

faidon closed T199251: furud: disconnect and power down all disk shelves as Resolved.
Wed, Oct 30, 11:33 PM · ops-codfw, Operations, DC-Ops

Tue, Oct 29

faidon added a comment to T180641: reinstall RT server with private IP and Buster.

@faidon RT is fixed Login works again.

Tue, Oct 29, 9:57 AM · Operations

Mon, Oct 28

faidon added a comment to T180641: reinstall RT server with private IP and Buster.

RT seems to be currently pointed at moscovium, and is currently broken: the frontpage doesn't load properly (mixed content messages) and the login doesn't work. If this can't be addressed quickly, perhaps we can rollback to the old host (ununpentium) in the meantime? Thanks!

Mon, Oct 28, 11:24 AM · Operations

Fri, Oct 25

faidon updated the task description for T174637: Setup esams atlas anchor.
Fri, Oct 25, 3:08 PM · Operations, netops, ops-esams
faidon added a comment to T223292: Netbox: generate CSV backups.

Ping! I'd like to start killing old entries from esams, but I'd like to make sure we have them backed up first.

Fri, Oct 25, 12:15 PM · netbox

Fri, Oct 18

faidon reopened T223292: Netbox: generate CSV backups as "Open".

It looks like this just dumps files flat, without keeping any archives. We've already lost a bunch of history unfortunately :(

Fri, Oct 18, 11:59 AM · netbox

Thu, Oct 17

faidon closed T235716: update librenms report as Resolved.

Fixed.

Thu, Oct 17, 2:16 PM · Operations, netbox
faidon changed the status of T225121: (Need By: Sept 30) upgrade msw1-eqiad from EX4200 to EX4300 from Stalled to Open.

What's the status of this? It seems like this migration is in some limbo state :)

Thu, Oct 17, 8:56 AM · netops, Operations, ops-eqiad

Oct 3 2019

faidon committed rOHPU12b82a8edc9e: Add new labsdb to cloud filter (authored by ayounsi).
Add new labsdb to cloud filter
Oct 3 2019, 11:11 AM
faidon committed rOHPU11702d17194d: Add BGP POPs to COREs export policy (authored by ayounsi).
Add BGP POPs to COREs export policy
Oct 3 2019, 11:11 AM
faidon committed rOHPUd85bf7dd03d9: Remove unit number from the LLDP config (authored by ayounsi).
Remove unit number from the LLDP config
Oct 3 2019, 11:11 AM
faidon committed rOHPU74d538482101: eqsin NTT traffic engineering (authored by ayounsi).
eqsin NTT traffic engineering
Oct 3 2019, 11:11 AM
faidon committed rOHPUcfa303ecdbbd: Add jbond superuser (authored by ayounsi).
Add jbond superuser
Oct 3 2019, 11:11 AM
faidon committed rOHPU133cc63e9b09: Add wdqs.svc.eqiad to analytics term wdqs (authored by ayounsi).
Add wdqs.svc.eqiad to analytics term wdqs
Oct 3 2019, 11:11 AM
faidon committed rOHPU66d52721b247: Add port 9105 to term prometheus in filter labs-in4 (authored by ayounsi).
Add port 9105 to term prometheus in filter labs-in4
Oct 3 2019, 11:11 AM
faidon committed rOHPU4a3704bdc6c5: Remove policy BGP_Wikimedia_own_space replace with BGP_Wikimedia_no_dfz (authored by ayounsi).
Remove policy BGP_Wikimedia_own_space replace with BGP_Wikimedia_no_dfz
Oct 3 2019, 11:11 AM
faidon committed rOHPUe26faf3e6bdb: Use anycast DNS IP (authored by ayounsi).
Use anycast DNS IP
Oct 3 2019, 11:11 AM
faidon committed rOHPU78088fc1a5b6: Set DNS and syslog targets to their anycast version (authored by ayounsi).
Set DNS and syslog targets to their anycast version
Oct 3 2019, 11:11 AM
faidon committed rOHPU055225ad5f67: Add analytics eventgate term (authored by ayounsi).
Add analytics eventgate term
Oct 3 2019, 11:11 AM
faidon committed rOHPU61fe84121335: Fix typo (missing /32 behind ACL IP) (authored by ayounsi).
Fix typo (missing /32 behind ACL IP)
Oct 3 2019, 11:11 AM
faidon committed rOHPU382b8e7a8d22: Add ldap-ro to Analytics ldap term (authored by ayounsi).
Add ldap-ro to Analytics ldap term
Oct 3 2019, 11:11 AM
faidon committed rOHPUfef92cf1cdd5: Reject RPKI invalid on IXP/Private peering sessions (authored by ayounsi).
Reject RPKI invalid on IXP/Private peering sessions
Oct 3 2019, 11:11 AM
faidon committed rOHPUcfae29ad1da5: Add term allow-anycast-dns in filter labs-in4 (authored by ayounsi).
Add term allow-anycast-dns in filter labs-in4
Oct 3 2019, 11:11 AM
faidon committed rOHPUd1d871a22245: Add centrallog1001 to syslog firewall terms (authored by ayounsi).
Add centrallog1001 to syslog firewall terms
Oct 3 2019, 11:11 AM
faidon committed rOHPU21e2c6dca282: Remove Casey (authored by ayounsi).
Remove Casey
Oct 3 2019, 11:11 AM
faidon committed rOHPUb09be4f8c0f9: Add RPKI classification (authored by ayounsi).
Add RPKI classification
Oct 3 2019, 11:11 AM
faidon committed rOHPU86ead89adb58: Add udp to analytics kerberos acl (authored by ayounsi).
Add udp to analytics kerberos acl
Oct 3 2019, 11:11 AM
faidon committed rOHPUdcb8249deb61: shrink 91.198.174.224/28 to 91.198.174.232/29 (authored by ayounsi).
shrink 91.198.174.224/28 to 91.198.174.232/29
Oct 3 2019, 11:11 AM
faidon committed rOHPUf907f27be45d: Add kpasswd to kerberos/analytics term (authored by ayounsi).
Add kpasswd to kerberos/analytics term
Oct 3 2019, 11:11 AM
faidon committed rOHPUde5bc3a7481c: Replace logstash target with per DC collector (authored by ayounsi).
Replace logstash target with per DC collector
Oct 3 2019, 11:11 AM
faidon committed rOHPUce5330f74315: Add mr1-eqsin (authored by ayounsi).
Add mr1-eqsin
Oct 3 2019, 11:11 AM
faidon committed rOHPUa76d42435c46: Add filter cloud-in4 to codfw (authored by ayounsi).
Add filter cloud-in4 to codfw
Oct 3 2019, 11:11 AM
faidon committed rOHPU0e6d06196376: Add icmp frag discard v4 terms (authored by ayounsi).
Add icmp frag discard v4 terms
Oct 3 2019, 11:11 AM
faidon committed rOHPU1d3f461eeb7e: Add analytics term kerberos (authored by ayounsi).
Add analytics term kerberos
Oct 3 2019, 11:11 AM
faidon committed rOHPU08768b05d3af: Make analytics-in6 term match routers (authored by ayounsi).
Make analytics-in6 term match routers
Oct 3 2019, 11:11 AM
faidon committed rOHPUcaf7b4f0ba51: Add routing-options section and RPKI validation (authored by ayounsi).
Add routing-options section and RPKI validation
Oct 3 2019, 11:11 AM
faidon committed rOHPU0aec928389d1: Make cloud-in4 term match routers (authored by ayounsi).
Make cloud-in4 term match routers
Oct 3 2019, 11:11 AM
faidon committed rOHPUfe77fa8bfc1c: Make analytics-in4 term match routers (authored by ayounsi).
Make analytics-in4 term match routers
Oct 3 2019, 11:11 AM
faidon committed rOHPU3d94cafa868b: msw: add config option to disable LLDP on some ports (authored by ayounsi).
msw: add config option to disable LLDP on some ports
Oct 3 2019, 11:11 AM
faidon committed rOHPU7a4c673153a8: Support more (incl. outbound) PCCW communities (authored by ayounsi).
Support more (incl. outbound) PCCW communities
Oct 3 2019, 11:11 AM
faidon committed rOHPU022482c1e158: Add ping offload (authored by ayounsi).
Add ping offload
Oct 3 2019, 11:11 AM
faidon committed rOHPU9f98cc87f1f0: Add eqsin routing specifics (authored by ayounsi).
Add eqsin routing specifics
Oct 3 2019, 11:11 AM
faidon committed rOHPU26ceed82e275: Remove sandbox-out6 filter (authored by ayounsi).
Remove sandbox-out6 filter
Oct 3 2019, 11:11 AM
faidon committed rOHPU9689225175d4: Remove decom eqiad switches, add asw1-eqsin (authored by ayounsi).
Remove decom eqiad switches, add asw1-eqsin
Oct 3 2019, 11:11 AM
faidon committed rOHPUf98a0a90e6ea: Add VC license keys (authored by ayounsi).
Add VC license keys
Oct 3 2019, 11:11 AM
faidon committed rOHPUd3ad51064620: Fix msw template's to match running conf (authored by ayounsi).
Fix msw template's to match running conf
Oct 3 2019, 11:11 AM
faidon committed rOHPU8c16052a0aea: Add mutihop bfd term to loopback4 (authored by ayounsi).
Add mutihop bfd term to loopback4
Oct 3 2019, 11:11 AM
faidon committed rOHPU03a766e293af: Add labsdb1012 to analytics mysql term (authored by ayounsi).
Add labsdb1012 to analytics mysql term
Oct 3 2019, 11:11 AM
faidon committed rOHPU847bbad1f79a: Remove terms peer-private-peer and peer from BGP_community_actions (authored by ayounsi).
Remove terms peer-private-peer and peer from BGP_community_actions
Oct 3 2019, 11:11 AM
faidon committed rOHPU433b24fcc6ef: Update analytics-in4 term mysql-dbstore (authored by ayounsi).
Update analytics-in4 term mysql-dbstore
Oct 3 2019, 11:11 AM
faidon committed rOHPU17e59ece8e86: Add cr2-eqsin (authored by ayounsi).
Add cr2-eqsin
Oct 3 2019, 11:11 AM
faidon committed rOHPUbb4d172b93b2: Update cloud-in4 filter (authored by ayounsi).
Update cloud-in4 filter
Oct 3 2019, 11:11 AM
faidon committed rOHPU19da68071eb9: Firewall: analytics-in4 mysql-dbstore + cloud transport renumbering (authored by ayounsi).
Firewall: analytics-in4 mysql-dbstore + cloud transport renumbering
Oct 3 2019, 11:11 AM
faidon committed rOHPUbcc95c63bc1a: Remove peering4/6 prefix-list (authored by ayounsi).
Remove peering4/6 prefix-list
Oct 3 2019, 11:11 AM
faidon committed rOHPU52292aaeb6fb: Add maintenance and rollback to the operations class (authored by ayounsi).
Add maintenance and rollback to the operations class
Oct 3 2019, 11:11 AM
faidon committed rOHPU0852e14e99d7: Remove filter sandbox-out4 (authored by ayounsi).
Remove filter sandbox-out4
Oct 3 2019, 11:11 AM
faidon committed rOHPU2da40eb3fc72: Ignore HE routes if learned from RS (dfw,ams,eqiad) (authored by ayounsi).
Ignore HE routes if learned from RS (dfw,ams,eqiad)
Oct 3 2019, 11:11 AM
faidon committed rOHPU5f28f49da77d: Add term return-tcp to loopback filters (authored by ayounsi).
Add term return-tcp to loopback filters
Oct 3 2019, 11:11 AM
faidon committed rOHPU0b51e1de36b9: BGP_transit_in, BGP_IXP_in and BGP_Private_Peer_in changes (authored by ayounsi).
BGP_transit_in, BGP_IXP_in and BGP_Private_Peer_in changes
Oct 3 2019, 11:11 AM
faidon committed rOHPU94a264358ac1: Add BGP_IXP_RS_in policy-statement (authored by ayounsi).
Add BGP_IXP_RS_in policy-statement
Oct 3 2019, 11:11 AM
faidon committed rOHPU67090e5921f3: Add missing eqsin avoid-path to jnt (authored by ayounsi).
Add missing eqsin avoid-path to jnt
Oct 3 2019, 11:11 AM
faidon committed rOHPUe2501baf662b: Remove old avoid-paths (authored by ayounsi).
Remove old avoid-paths
Oct 3 2019, 11:11 AM
faidon committed rOHPU040dc43ae5a2: Comment out cr3-esams (authored by ayounsi).
Comment out cr3-esams
Oct 3 2019, 11:11 AM
faidon committed rOHPU806f98ed5987: Remove neodymium/sarin from labs-in4 + add cumin1001 (authored by ayounsi).
Remove neodymium/sarin from labs-in4 + add cumin1001
Oct 3 2019, 11:11 AM
faidon committed rOHPU69e4254a496c: Grow frack codfw private range to 10.195.0.0/24 (authored by ayounsi).
Grow frack codfw private range to 10.195.0.0/24
Oct 3 2019, 11:11 AM
faidon committed rOHPU3f14029cd23f: New ulsfo/eqord/eqdfw devices (authored by ayounsi).
New ulsfo/eqord/eqdfw devices
Oct 3 2019, 11:11 AM
faidon committed rOHPUf6bd3a867032: Add support for Junos mgmt routing instance (authored by ayounsi).
Add support for Junos mgmt routing instance
Oct 3 2019, 11:11 AM
faidon committed rOHPU8722ac305823: Add term labnet-nova-api to cloud-in4 (authored by ayounsi).
Add term labnet-nova-api to cloud-in4
Oct 3 2019, 11:11 AM
faidon committed rOHPU85389bc530de: Add volans account (authored by ayounsi).
Add volans account
Oct 3 2019, 11:11 AM