fgiunchedi (Filippo Giunchedi)
Awesome

Projects (19)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 8:06 AM (128 w, 6 d)
Availability
Available
IRC Nick
godog
LDAP User
Filippo Giunchedi
MediaWiki User
Filippo Giunchedi

Recent Activity

Yesterday

fgiunchedi claimed T161193: Upgrade to Grafana 4.2.0.

@Peter yeah I can, standard procedure is to upgrade https://grafana-labs.wikimedia.org first and see how that goes, if no issues arise then upgrade https://grafana.wikimedia.org too.

Thu, Mar 23, 3:54 PM · User-fgiunchedi, Performance-Team, Operations
fgiunchedi moved T143405: Move labs 'instances' data to graphite labs from Backlog to Blocked on the User-fgiunchedi board.
Thu, Mar 23, 11:48 AM · User-fgiunchedi, Patch-For-Review, Labs, Operations, Graphite
fgiunchedi moved T160677: Effects on adjusting Prometheus retention from Backlog to Doing on the User-fgiunchedi board.
Thu, Mar 23, 11:48 AM · User-fgiunchedi, Operations, Prometheus-metrics-monitoring
fgiunchedi closed T152504: rack/setup prometheus100[3-4] as "Resolved".

Finishing the setup in T148408: Put prometheus baremetal servers in service

Thu, Mar 23, 11:47 AM · User-fgiunchedi, Patch-For-Review, ops-eqiad, Operations

Wed, Mar 22

fgiunchedi closed T160644: Eventstreams graphite disk usage as "Resolved".

eventstreams is at 110G and cleaned up periodically, good enough for now

Wed, Mar 22, 7:17 PM · Patch-For-Review, Analytics
fgiunchedi closed T128796: rack new mw log host - sinistra as "Declined".

@Dzahn @Krinkle yeah sinistra was reinstalled as mwlog2001 in T153384: reinstall/reimage sinistra as mwlog2001

Wed, Mar 22, 9:30 AM · Patch-For-Review, DC-Ops, codfw-rollout-Jan-Mar-2016, codfw-rollout, ops-codfw, Operations
fgiunchedi closed T128796: rack new mw log host - sinistra, a subtask of T126988: Log host for codfw (fluorine's equivalent), as "Declined".
Wed, Mar 22, 9:30 AM · hardware-requests, Operations, codfw-rollout-Jan-Mar-2016, codfw-rollout
fgiunchedi triaged T160640: Rack and Setup ms-be1028-ms-1039 as "High" priority.

@Cmjohnson is there an ETA to have the servers with OS installed online? thanks!

Wed, Mar 22, 9:09 AM · Patch-For-Review, User-fgiunchedi, Operations, ops-eqiad

Tue, Mar 21

fgiunchedi added a comment to T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU.

@hashar indeed the workers 100 is static in puppet, I'm not going to have time for it but please feel free to play with the setting and see if that makes a difference!

Tue, Mar 21, 5:27 PM · Continuous-Integration-Infrastructure (Little Steps Sprint), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
fgiunchedi added a project to T150734: Make Thumbor logs available in ELK: User-fgiunchedi.

ok @Gilles ! I'll review/upload likely next week

Tue, Mar 21, 5:26 PM · User-fgiunchedi, Patch-For-Review, Performance-Team, Thumbor
fgiunchedi moved T160986: Decommission ms-fe100[1-4] from Backlog to Blocked on the User-fgiunchedi board.
Tue, Mar 21, 10:40 AM · User-fgiunchedi, hardware-requests, Operations
fgiunchedi added a project to T160986: Decommission ms-fe100[1-4]: User-fgiunchedi.
Tue, Mar 21, 10:40 AM · User-fgiunchedi, hardware-requests, Operations
fgiunchedi closed T155095: Rack and set up ms-fe100[5-8] as "Resolved".

Done

Tue, Mar 21, 10:38 AM · User-fgiunchedi, Patch-For-Review, Operations, ops-eqiad
fgiunchedi created T160986: Decommission ms-fe100[1-4].
Tue, Mar 21, 10:38 AM · User-fgiunchedi, hardware-requests, Operations
fgiunchedi moved T125020: upgrade netmon1001 to jessie from Backlog to Doing on the User-fgiunchedi board.
Tue, Mar 21, 10:07 AM · Patch-For-Review, User-fgiunchedi, Operations
fgiunchedi claimed T125020: upgrade netmon1001 to jessie.
Tue, Mar 21, 10:07 AM · Patch-For-Review, User-fgiunchedi, Operations

Mon, Mar 20

fgiunchedi closed T160736: Requesting access to perf-roots for gilles as "Resolved".

Done

Mon, Mar 20, 5:52 PM · Operations, Ops-Access-Requests
fgiunchedi added a comment to T160736: Requesting access to perf-roots for gilles.

Discussion on today's ops meeting result: approved

Mon, Mar 20, 5:01 PM · Operations, Ops-Access-Requests
fgiunchedi moved T160640: Rack and Setup ms-be1028-ms-1039 from Backlog to Blocked on the User-fgiunchedi board.
Mon, Mar 20, 10:21 AM · Patch-For-Review, User-fgiunchedi, Operations, ops-eqiad
fgiunchedi updated subscribers of T160738: Increase Grafana user rights for Performance team members.

IIRC grafana permissions are not puppetized :( @Gilles you can delete alerts though? I see @Krinkle is an admin on grafana so maybe that's part of the discrepancy

Mon, Mar 20, 10:17 AM · Performance-Team, Operations, Graphite

Thu, Mar 16

fgiunchedi added a project to T150479: Error collecting metrics from varnish_exporter on some misc hosts: User-fgiunchedi.
Thu, Mar 16, 7:07 PM · User-fgiunchedi, Patch-For-Review, Traffic, Prometheus-metrics-monitoring, Operations
fgiunchedi edited the description of T160677: Effects on adjusting Prometheus retention.
Thu, Mar 16, 7:01 PM · User-fgiunchedi, Operations, Prometheus-metrics-monitoring
fgiunchedi created T160677: Effects on adjusting Prometheus retention.
Thu, Mar 16, 6:21 PM · User-fgiunchedi, Operations, Prometheus-metrics-monitoring
fgiunchedi added a comment to T160644: Eventstreams graphite disk usage.

Yar, this is because many of the metrics are per-client. I'd like to know if clients start lagging, and there's not a real way to aggregate that.

But, we really don't need to keep history of this data. Can we delete certain data > 2 weeks old?

Thu, Mar 16, 6:10 PM · Patch-For-Review, Analytics
fgiunchedi added a comment to T160640: Rack and Setup ms-be1028-ms-1039.

reporting from IRC

Thu, Mar 16, 6:03 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-eqiad
fgiunchedi edited the description of T160640: Rack and Setup ms-be1028-ms-1039.
Thu, Mar 16, 5:21 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-eqiad
fgiunchedi added a project to T160640: Rack and Setup ms-be1028-ms-1039: User-fgiunchedi.

racking will be 3x systems per row, 10G where possible or 1G where we can't do 10G (see also T148647)

Thu, Mar 16, 5:21 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-eqiad
fgiunchedi moved T152504: rack/setup prometheus100[3-4] from Backlog to Doing on the User-fgiunchedi board.
Thu, Mar 16, 3:59 PM · User-fgiunchedi, Patch-For-Review, ops-eqiad, Operations
fgiunchedi edited the description of T160644: Eventstreams graphite disk usage.
Thu, Mar 16, 3:57 PM · Patch-For-Review, Analytics
fgiunchedi created T160644: Eventstreams graphite disk usage.
Thu, Mar 16, 3:56 PM · Patch-For-Review, Analytics
fgiunchedi added a project to T143405: Move labs 'instances' data to graphite labs: User-fgiunchedi.
Thu, Mar 16, 3:50 PM · User-fgiunchedi, Patch-For-Review, Labs, Operations, Graphite
fgiunchedi placed T143405: Move labs 'instances' data to graphite labs up for grabs.

I can't work on this now, though instances is taking 165G on production graphite now. It'd be nice to have it moved to labmon instead, @Andrew @madhuvishy @yuvipanda @chasemp

Thu, Mar 16, 3:50 PM · User-fgiunchedi, Patch-For-Review, Labs, Operations, Graphite
fgiunchedi changed the status of T147923: Extract metrics from logs from "Stalled" to "Open".

Upstream patch has been merged, though there's another outstanding issue https://github.com/google/mtail/issues/56 which makes mtail crash on parse errors. We've also seen mtail stopping sending metrics after a while

Thu, Mar 16, 9:54 AM · User-fgiunchedi, Patch-For-Review, Monitoring, Operations
fgiunchedi added a project to T136312: encrypt syslog traffic: User-fgiunchedi.
Thu, Mar 16, 9:16 AM · User-fgiunchedi, Operations
fgiunchedi created T160616: Enable HTTPS for swift clients.
Thu, Mar 16, 9:15 AM · Traffic, HTTPS, User-fgiunchedi, Operations
fgiunchedi accepted D599: The engine is responsible for deleting the source temp file no matter what.
Thu, Mar 16, 9:02 AM

Wed, Mar 15

fgiunchedi placed T112257: rename cassandra cluster up for grabs.
Wed, Mar 15, 1:01 PM · Services (watching), Operations, Patch-For-Review, RESTBase, RESTBase-Cassandra
fgiunchedi added a project to T151009: Provide authenticated access to Prometheus native web interface: User-fgiunchedi.
Wed, Mar 15, 1:00 PM · User-fgiunchedi, Operations, Prometheus-metrics-monitoring
fgiunchedi placed T125140: File not found after reupload up for grabs.

Indeed the previous version from 2007 doesn't seem to be there, not sure there's much we can do

Wed, Mar 15, 1:00 PM · Operations, media-storage
fgiunchedi closed T129196: mw2212 had several downtimes recently - test before repool as "Resolved".

No issues, resolving

Wed, Mar 15, 12:53 PM · DC-Ops, Patch-For-Review, ops-codfw, Operations
fgiunchedi closed T150680: rack/setup restbase201[0-2] as "Resolved".

This was done a while ago

Wed, Mar 15, 12:50 PM · Patch-For-Review, Operations, ops-codfw
fgiunchedi added a project to T152504: rack/setup prometheus100[3-4]: User-fgiunchedi.
Wed, Mar 15, 12:50 PM · User-fgiunchedi, Patch-For-Review, ops-eqiad, Operations
fgiunchedi added a project to T158286: Raise default logging level of prometheus-hhvm-exporter: User-fgiunchedi.
Wed, Mar 15, 12:49 PM · User-fgiunchedi, HHVM, Prometheus-metrics-monitoring
fgiunchedi reassigned T155690: troubleshoot drac on ms-be2010.codfw.wmnet from fgiunchedi to RobH.
Wed, Mar 15, 12:49 PM · ops-codfw, Operations
fgiunchedi accepted D587: Add ability to throttle failing thumbnails.

Thanks @Gilles for the explanation!
I was thinking about memcache connection pooling, though each thumbor instance will have its own connection(s). If it becomes a problem we can put sth like nutcracker on localhost

Wed, Mar 15, 9:15 AM · User-fgiunchedi
fgiunchedi added a project to D587: Add ability to throttle failing thumbnails: User-fgiunchedi.
Wed, Mar 15, 9:12 AM · User-fgiunchedi
fgiunchedi moved T160488: Degraded RAID on ms-be1008 from Backlog to Blocked on the User-fgiunchedi board.
Wed, Mar 15, 9:11 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi added a project to T160488: Degraded RAID on ms-be1008: User-fgiunchedi.
Wed, Mar 15, 9:05 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi assigned T160488: Degraded RAID on ms-be1008 to Cmjohnson.

@Cmjohnson if we have 2TB spares onsite please replace. We can reclaim the disks once decommission happens in a few weeks

Wed, Mar 15, 9:05 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi edited the description of T160488: Degraded RAID on ms-be1008.
Wed, Mar 15, 9:04 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi moved T148408: Put prometheus baremetal servers in service from Backlog to Doing on the User-fgiunchedi board.
Wed, Mar 15, 8:28 AM · User-fgiunchedi, Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi moved T155095: Rack and set up ms-fe100[5-8] from Backlog to Doing on the User-fgiunchedi board.
Wed, Mar 15, 8:28 AM · User-fgiunchedi, Patch-For-Review, Operations, ops-eqiad
fgiunchedi added a project to T149451: Get 5xx logs into kibana/logstash: User-fgiunchedi.
Wed, Mar 15, 8:28 AM · User-fgiunchedi, Wikimedia-Logstash, Operations
fgiunchedi added a project to T125020: upgrade netmon1001 to jessie: User-fgiunchedi.
Wed, Mar 15, 8:28 AM · Patch-For-Review, User-fgiunchedi, Operations
fgiunchedi added a project to T148408: Put prometheus baremetal servers in service: User-fgiunchedi.
Wed, Mar 15, 8:27 AM · User-fgiunchedi, Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi added a project to T155095: Rack and set up ms-fe100[5-8]: User-fgiunchedi.
Wed, Mar 15, 8:27 AM · User-fgiunchedi, Patch-For-Review, Operations, ops-eqiad
fgiunchedi moved T148541: Evaluate prometheus snmp_exporter for Torrus PDUs metrics use case from Backlog to Doing on the User-fgiunchedi board.
Wed, Mar 15, 8:26 AM · User-fgiunchedi, Patch-For-Review, Prometheus-metrics-monitoring, Monitoring, Operations
fgiunchedi added a project to T148541: Evaluate prometheus snmp_exporter for Torrus PDUs metrics use case: User-fgiunchedi.
Wed, Mar 15, 8:26 AM · User-fgiunchedi, Patch-For-Review, Prometheus-metrics-monitoring, Monitoring, Operations
fgiunchedi moved T158337: codfw: ms-be2028-ms-be2039 rack/setup from Backlog to Doing on the User-fgiunchedi board.
Wed, Mar 15, 8:26 AM · User-fgiunchedi, Patch-For-Review, Operations, ops-codfw
fgiunchedi added a project to T158337: codfw: ms-be2028-ms-be2039 rack/setup: User-fgiunchedi.
Wed, Mar 15, 8:26 AM · User-fgiunchedi, Patch-For-Review, Operations, ops-codfw

Tue, Mar 14

fgiunchedi added a comment to T155689: ms-be2002.codfw.wmnet has drac issues.

FWIW this host is slated for decom in some weeks, I wouldn't spend too much time on its idrac especially if there's other hosts not to be decom with broken idrac

Tue, Mar 14, 3:38 PM · ops-codfw, Operations
fgiunchedi created T160419: Request for project "User-fgiunchedi" for organizing my personal workboard.
Tue, Mar 14, 3:15 PM · Project-Admins
fgiunchedi closed T160312: Degraded RAID on ms-be2008 as "Resolved".

Disk rebuilding

Tue, Mar 14, 3:01 PM · Operations, ops-codfw

Mon, Mar 13

fgiunchedi assigned T160312: Degraded RAID on ms-be2008 to Papaul.

@Papaul ok! please replace even though we're decommissioning the machines in 4-5 weeks, on the basis that disks will be wiped and possibly used as spares?

Mon, Mar 13, 6:08 PM · Operations, ops-codfw
fgiunchedi closed T160349: Degraded RAID on ms-be2028 as "Resolved".

My mistake, host is provisioning

Mon, Mar 13, 3:09 PM · Operations, ops-codfw
fgiunchedi closed T127455: Enable HTTPS for Swift traffic as "Resolved".

HTTPS for ms-fe.svc is now active in eqiad and codfw

Mon, Mar 13, 1:34 PM · Patch-For-Review, Wikimedia-Multiple-active-datacenters, media-storage, Operations
fgiunchedi closed T127455: Enable HTTPS for Swift traffic, a subtask of T88445: MediaWiki active/active datacenter investigation and work, as "Resolved".
Mon, Mar 13, 1:33 PM · Performance-Team, Availability, Epic
fgiunchedi closed T127455: Enable HTTPS for Swift traffic, a subtask of T111653: Encrypt all the things, as "Resolved".
Mon, Mar 13, 1:33 PM · Operations
fgiunchedi updated subscribers of T160312: Degraded RAID on ms-be2008.

sde is unhappy

Mon, Mar 13, 11:22 AM · Operations, ops-codfw

Fri, Mar 10

fgiunchedi accepted D593: Clean up temp file on success in IM engine.
Fri, Mar 10, 4:22 PM
fgiunchedi created T160156: Add node_exporter ipvs ipv6 support.
Fri, Mar 10, 8:56 AM · Traffic, Operations, Monitoring

Thu, Mar 9

Dzahn awarded T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie) a Barnstar token.
Thu, Mar 9, 7:19 PM · Patch-For-Review, Operations
fgiunchedi added inline comments to D587: Add ability to throttle failing thumbnails.
Thu, Mar 9, 7:16 PM · User-fgiunchedi
fgiunchedi added a comment to T159830: Sanity check global-multiwrite logs for ConfirmEdit usage.

I took a quick look at the swift logs on lithium, all DELETEs seem to be successful (i.e. HTTP 200s) with the exception of some for which swift replied 404. All around 02.00 UTC in march. Does that help? Also re: deletion of all files but one, meaning one was left behind for no reason or deletions are happening for no reason?

Thu, Mar 9, 6:36 PM · Operations, media-storage
fgiunchedi closed T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie) as "Resolved".

This is completed, I've left out the part about sending logs to both datacenters as out of scope for this. The real solution as suggested by ottomata and tracked in T126989: MediaWiki logging & encryption is to use kafka for log shipping, that'd also buy us encryption

Thu, Mar 9, 5:10 PM · Patch-For-Review, Operations
fgiunchedi closed T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie), a subtask of T123525: reduce amount of remaining Ubuntu 12.04 (precise) systems in production, as "Resolved".
Thu, Mar 9, 5:10 PM · Goal, Operations
fgiunchedi closed T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie), a subtask of T156937: Understand the preparedness of misc services for datacenter switchover, as "Resolved".
Thu, Mar 9, 5:10 PM · Release-Engineering-Team, DC-Switchover-Prep-Q3-2016-17, Operations
fgiunchedi edited the description of T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie).
Thu, Mar 9, 5:09 PM · Patch-For-Review, Operations
fgiunchedi added a comment to T160082: codfw: oresrdb2002 rack/setup.

Row C works (oresrdb2001 is row B)

Thu, Mar 9, 4:44 PM · Patch-For-Review, Operations, ops-codfw

Wed, Mar 8

fgiunchedi added a comment to T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie).

Note mwlog[12]001 need to be whitelisted to be reachable from analytics vlan, rsync-http-https term

Wed, Mar 8, 10:12 AM · Patch-For-Review, Operations

Tue, Mar 7

fgiunchedi closed T149098: Experiment with Swift as docker registry backend as "Resolved".

We're using codfw to test swift as docker registry backend.

Tue, Mar 7, 4:36 PM · Patch-For-Review, Operations, User-Joe, Prod-Kubernetes, Kubernetes-production-experiment
fgiunchedi closed T149098: Experiment with Swift as docker registry backend, a subtask of T148960: Install a docker registry for production, as "Resolved".
Tue, Mar 7, 4:36 PM · Operations, User-Joe, Prod-Kubernetes, Kubernetes-production-experiment

Mon, Mar 6

fgiunchedi added a comment to T145072: Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb.

Update from the monitoring meeting, this can be implemented via puppetdb queries, additionally there's now classes to generate Prometheus config files e.g. with all hosts having a specific class and parameters applied. prometheus::class_config

Mon, Mar 6, 6:29 PM · DBA, Operations, Prometheus-metrics-monitoring

Sat, Mar 4

Milimetric awarded T154558: Periodic 500s from piwik.wikimedia.org a Barnstar token.
Sat, Mar 4, 3:35 AM · Wikipedia-iOS-App-Backlog, Patch-For-Review, Traffic, User-Elukey, Analytics-Kanban, Operations

Fri, Mar 3

fgiunchedi merged task T159540: Degraded RAID on ms-be1012 into T157237: Degraded RAID on ms-be1012.
Fri, Mar 3, 4:04 PM · ops-eqiad, Operations
fgiunchedi merged T159540: Degraded RAID on ms-be1012 into T157237: Degraded RAID on ms-be1012.
Fri, Mar 3, 4:04 PM · Patch-For-Review, ops-eqiad, Operations

Thu, Mar 2

fgiunchedi added a comment to T159413: Decomission ms-fe2001-4.

@Papaul I've effectively decomissioned the systems from production and they are still in puppet but running the spare role in puppet, I've marked as done the steps @RobH outlined.

Thu, Mar 2, 4:41 PM · hardware-requests, ops-codfw, Operations
fgiunchedi edited the description of T159413: Decomission ms-fe2001-4.
Thu, Mar 2, 4:40 PM · hardware-requests, ops-codfw, Operations
fgiunchedi renamed T155095: Rack and set up ms-fe100[5-8] from "Rack and set up ms-fe100[5-7]" to "Rack and set up ms-fe100[5-8]".
Thu, Mar 2, 3:03 PM · User-fgiunchedi, Patch-For-Review, Operations, ops-eqiad
fgiunchedi added a comment to T139372: Set up oresrdb redis node in codfw.

To get started with oresrdb2001 in codfw please see/review https://gerrit.wikimedia.org/r/#/c/34048

Thu, Mar 2, 3:02 PM · Patch-For-Review, Operations, Revision-Scoring-As-A-Service-Backlog
fgiunchedi closed T127762: Update Debian Package for Scap3 as "Resolved".

Package is out

Thu, Mar 2, 2:57 PM · Patch-For-Review, Deployment-Systems, Scap
fgiunchedi merged task T154250: rack/setup/install/track new ms-fe1005-1008 into T155095: Rack and set up ms-fe100[5-8].
Thu, Mar 2, 12:56 PM · ops-eqiad, Operations
fgiunchedi merged T154250: rack/setup/install/track new ms-fe1005-1008 into T155095: Rack and set up ms-fe100[5-8].
Thu, Mar 2, 12:56 PM · User-fgiunchedi, Patch-For-Review, Operations, ops-eqiad
fgiunchedi closed T152580: rollout prometheus-node-exporter 0.13 as "Resolved".

Resolving since tools has been upgraded as well.

Thu, Mar 2, 12:48 PM · Prometheus-metrics-monitoring
fgiunchedi added a comment to T152580: rollout prometheus-node-exporter 0.13.

@scfc that's awesome, thanks a lot for your help!

Thu, Mar 2, 12:47 PM · Prometheus-metrics-monitoring
fgiunchedi closed T158633: .upper .lower .mean metrics are off when looking at the last 7 days datapoints of a Nodepool time metric as "Resolved".

@hashar maybe the aggregation was different on graphite2001, that'd be my guess. Anyways we're going to reprovision grpahtie2001 this week so I'm tentatively resolving

Thu, Mar 2, 12:41 PM · Nodepool, Graphite
fgiunchedi closed T159437: swift-account-stats: Max retries exceeded with url: /auth/v1.0 as "Resolved".

I'm assuming this happened on ms-fe2001 today? It was during decomission and a legitimate error, I'll tentatively resolve

Thu, Mar 2, 12:00 PM · media-storage, Operations
fgiunchedi closed T159437: swift-account-stats: Max retries exceeded with url: /auth/v1.0, a subtask of T132324: Tracking and Reducing cron-spam from root@ , as "Resolved".
Thu, Mar 2, 12:00 PM · Patch-For-Review, User-Elukey, Operations
fgiunchedi created T159422: zero.wikimedia.org references bits.wikimedia.org.
Thu, Mar 2, 10:01 AM · ZeroPortal, Zero
fgiunchedi reassigned T159413: Decomission ms-fe2001-4 from fgiunchedi to Papaul.

All four machines ms-fe200[1-4] are running spare::system role and can be decomissioned.

Thu, Mar 2, 9:36 AM · hardware-requests, ops-codfw, Operations