Consolidate performance website and related software
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Feb 23 2017, 9:08 AM

Description

When moving graphite1001 to jessie I noticed the performance website is also hosted there, together with coal, xenon is instead hosted on fluorine (now mwlog1001), xhgui and its mongodb are on tungsten. Since planning for capex/opex next year is on, I'm proposing to consolidate all performance-related software onto VM or baremetal depending on required specs. Thoughts?

Sub tasks

Write up understanding of the current (scattered) services. (Below: Current services)
Figure out overall migration plan in terms of hardware. – (Below: Proposed topology)
New stack:
- Request production VM for webperf (metrics processing). – T179036
- Migrate webperf from hafnium to webperf#001. – T186774
- Figure out how to migrate performance.wikimedia.org site.
- Migrate coal from graphite1001 to webperf#001. – T159354
- Request production VM for web apps (xhgui, xenon). - T194390
- make xhgui::app role support stretch/buster and deploy on new xhgui machines - T238788
- Migrate xhghui from tungsten to new VM. - VMs created in T238098
- Update wmf-config to write XWD profiles to new XHGUI location.
- Figure out how to migrate Xenon processing. (See "Moveability" below)
- Move Xenon's ArcLamp and Apache-for-xenon from mwlog1001 to webperf1002.
Old stack:
- Decom webperf/asset-check service. – T164419
- Decom webperf/ve service. – T175083
- Decom old xhprof viewer. – T196406
- Decom osmium.eqiad host. – T175093
- Decom hafnium.eqiad host. – T193420
- Decom tungsten.eqiad host. – T260395

Current services

performance.wikimedia.org: Static website. Proxied services: coal (local), xenon (mwlog1001), xhgui/xhprof (tungsten)
- Currently hosted on webperf1001.
- Code: puppet:/role/performance/site, puppet:../apache/sites/performance.wikimedia.org
- Code: performance/docroot.git
coal: EventLogging subscriber (Kafka) that processes Navigation Timing events and stores aggregated data into a dedicated Graphite backend).
- Currently on webperf1001.
- Code: performance/coal
- See T159354: Move coal from graphite#001 nodes to webperf#001
coal-web: Python http service that serves json-formatted data from the coal storage.
- Currently on webperf1001
- Uses the graphite HTTP API to pull raw data and to format that correctly. The http service is exposed via a private socket file, proxied from performance.wikimedia.org Apache config (currently on the same server).
- Code: performance/coal
xenon: Receives data from all app servers on a Redis instance (see operations/mediawiki-config.git:/StartProfiler). The xenon-log python service subscribes to this Redis feeds and produces searchable log files. A cron (xenon-generate-svgs) periodically produces SVGs which are stored in a local directory and made available on performance.wikimedia.org through a local Apache proxy.
- Aside from the published SVGs (which don't need to be on mwlog1001), these log files are also manually searchable through a command-line tool xenon-grep.
- Currently on mwlog1001.
- Code: puppet:/role/xenon, puppet:/modules/xenon, puppet:/files/xenon-log, ../apache/sites/xenon
xhgui: A webapp for viewing and analyzing PHP profiling data. Wikimedia-Debug requests can generate a profile that is submitted to the XHGui's MongoDB database. (StartProfiler).
- Currently on tungsten.
- Code: puppet:/role/xhgui.
webperf: EventLogging subscriber deamons that send data to Statsd/Graphite. webperf::statsv, webperf::ve, webperf::navtiming.
- Currently on hafnium.
- Code: puppet:/role/webperf

Moveability

In T158837#3349354, @Krinkle wrote:

performance.wikimedia.org:

Currently on graphite1001.

Should be trivial to move to another server.

coal and coal-web:

Currently on graphite1001.

Could be moved to a separate server if that separate server becomes a secondary graphite backend (like graphite1003). But I'd rather not make our perf/misc server a production graphite backend. So, unless we allocate two servers, probably makes sense to keep coal-web on graphite1001 for now.

xenon:

Currently on mwlog1001. 4 things: 1) A Redis server for incoming data from app servers, 2) Process to create text files from Redis data, 3) Process to create SVG files, 4) Apache to serve these files.

2, 3 and 4 are easy to move to another server. The Redis server, and therefore the part that is interacted with from production I'd prefer to keep on mwlog1001.

xhgui:

Currently on tungsten. Requires MongoDB, PHP, Apache.

Easily kept or moved, if another server would become the perf server.

webperf

Currently on hafnium.

Easily moved.

Old topology

Based on T158837#3368030:

graphite1001: (Our stuff is minor/secondary)

(Our) services: coal, coal-web, perf-site.

mwlog1001: (Our stuff is minor/secondary)

(Our) services: Redis (endpoint receiving xenon data), xenon-log (reads Redis, writes TXT), xenon-generate-svgs (reads TXT, writes SVG), Apache (serves TXT and SVG, proxied from perf-site)

tungsten (former db server; old, should be decom)

Spec: 16 cores, 64G RAM, 40G and 1.6T HHD
Services: XHGui (MongoDB, Apache)

osmium (former app server):

Spec: 16 cores, 64G RAM, 2x 500G HDD
Services: – (unused, previously: visualeditor, devwiki, jsbench)

hafnium (misc; old; should be decom/replaced):

Spec: 24 cores, 32G RAM, 50G HHD
Services: webperf (navtiming, statsv)

New topology

Based on T158837#3582514:

webperf x001 (Ganeti VM - multi-dc)

Specs: 4 vCPU, 8GB RAM, 50GB HHD
Services: webperf/processors_and_site (perf-site, coal::processor, coal::web, statsv, navtiming)

webperf x002 (Ganeti VM - multi-dc)

Specs: 4 vCPU, 8GB RAM, 50GB HHD
Services: webperf/profiling_tools (xhgui, arc-lamp, Apache for arc-lamp)

Details

Subject	Repo	Branch	Lines +/-
cumin: update xhgui alias to apply to new role name	operations/puppet	production	+1 -1
delete role::xhgui::app	operations/puppet	production	+2 -127
webperf: switch xhgui_host from tungsten to xhgui1001	operations/puppet	production	+1 -1
hieradata: Include cache-text in Beta Cluster 'cache_hosts'	operations/puppet	production	+1 -0
hieradata: Move beta 'cache::app_directors' from Horizon to Puppet	operations/puppet	production	+34 -0
xhgui: rsync mongodb data from tungsten to xhgui1001	operations/puppet	production	+9 -0
webperf: Remove arclamp subscriber from mwlog servers	operations/puppet	production	+3 -16
webperf: Add 'Server: <fqdn>' header to performance.wikimedia.org	operations/puppet	production	+4 -0
webperf: Rename webperf profiles for clarity	operations/puppet	production	+55 -45
Make links from php-profiling.html to to /xenon/ relative	performance/docroot	master	+5 -5
Need to install mongodb on xhgui machines	operations/puppet	production	+1 -0
webperf: Make the different webperf roles explicit	operations/puppet	production	+29 -24
performance website: remove from graphite hosts	operations/puppet	production	+2 -4
performance website: allow traffic	operations/puppet	production	+7 -0
performance.wikimedia.org: serve from webperfX001	operations/puppet	production	+5 -1
docroot: Pull graph data from graphite, not coal-web	performance/docroot	master	+134 -91
graphite: allow data requests from performance.wikimedia.org	operations/puppet	production	+2 -1
coal: revert coal-web back to what it was	performance/coal	master	+99 -105
coal: Point systemd and uwsgi config to scap-deployed version	operations/puppet	production	+28 -22
coal: convert to using graphite instead of writing to whisper	performance/coal	master	+113 -60
webperf: Move statsv monitoring from 'webperf' to 'webperf::statsv'	operations/puppet	production	+8 -7

Revisions and Commits

rOPUP Wikimedia Puppet
	rOPUPb6b72229db0d webperf: Remove unused deprecate.py
	rOPUPee96489a1607 webperf: Update navtiming.py to use eventlogging instead of zmq
	rOPUP35d45f78d7d4 webperf: Decom asset-check service
	rOPUPf982f9bcba05 webperf: Remove remnants of webperf::asset_check
	rOPUP6487a7948960 webperf: Use trebuchet install of eventlogging for ve.py
	rOPUPef40e5071cde webperf: Move statsv monitoring from 'webperf' to 'webperf::statsv'

Related Objects
Search...

Status	Assigned	Task
Resolved	• dpifke	T158837 Consolidate performance website and related software
Invalid	None	T159351 Investigate which xhgui pages are expensive and block them from public access
Resolved	• Imarlier	T159354 Move coal from graphite#001 nodes to webperf#001
Resolved	Dzahn	T194283 Clean up graphite1001.eqiad.wmnet, now that coal has been moved
Resolved	Krinkle	T161196 tungsten is out of space on /srv
Resolved	Krinkle	T164419 Turn off asset-check
Resolved	Krinkle	T175083 Decommission webperf/ve.py daemon
Resolved	• Cmjohnson	T175093 Decommission osmium.eqiad.wmnet
Resolved	Dzahn	T179036 Request VM for webperf (metrics processing)
Resolved	• Imarlier	T186774 Migrate webperf from hafnium to webperf1001
Resolved	• Cmjohnson	T193420 Decommission hafnium
Resolved	• Imarlier	T191994 Move coal and navtiming out of puppet
Resolved	Krinkle	T196406 Decom "xhprof" viewer
Resolved	Krinkle	T195312 Move flame graphs hosting from mwlog1001 to webperf-2 and enable in Beta Cluster
Resolved	Krinkle	T195314 Set up webperf-1 node in Beta Cluster
Resolved	• dpifke	T180761 Move XHGui from tungsten to xhgui-001
Resolved	Krinkle	T212468 Upgrade XHGui from 0.8.x to 0.9
Resolved	Dzahn	T238098 vm request for xhgui
Resolved	Dzahn	T238788 make xhgui::app role support stretch/buster and deploy on new xhgui machines
Resolved	Marostegui	T254795 Database for XHGui profiles
Resolved	Dzahn	T254310 Package XHGui as .deb
Resolved	Dzahn	T259278 Puppet failing due to dependency cycle on deployment-xhgui03
Resolved	Dzahn	T259206 reinstall xhgui* with buster
Resolved	akosiaris	T199853 Increase webperf1002/webperf2002 space from 50GB to 150GB (Ganeti)
Resolved	Jclark-ctr	T260395 decom tungsten

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

You should now be able to move xhgui over to webperf1002/2002. New VMs have been created and are ready to use. See details in T194390#4212203.

Change 433710 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] webperf: separate permissions from specific apps

https://gerrit.wikimedia.org/r/433710

@Imarlier @Krinkle I forgot about the shell access part. https://gerrit.wikimedia.org/r/#/c/433738/ added the perfteam admin group to the new VMs. You can now SSH to webperf1002 / webperf2002. I added the "perfteam" admin group but not the eventlogging-admin group so far.

@Imarlier I was just thinking about whether the etcd code is live or not for webperf/navtiming. The puppet invocation hasn't changed, but given navtiming.systemd uses the run_navtiming.py entry point (which reads config.ini), that presumably means the etcd_path/etcd_domain are used in prod. Is that right?

Anyhow, it currently uses MediaWiki's WMFMasterDatacenter key which seems inappropriate. It certainly seems desirable that, in the event of an overall switch-over that webperf/navtiming follows the other services, but we should also be able to switch over independently. Especially for navtiming, which would otherwise have downtime in the case of server maintenance.

I was discussing this with @Dzahn on IRC just now (who was asking whether it's fine to reboot the server to try enabling IPv6), which made me think it'd be nice if we could just switch over to codfw for a bit, upgrade the eqiad server, and then be able to switch back to try out, and switch back again if it doesn't work work. Instead of being subject to downtime.

Puppet has various inheritance options we can use to make our etcd config automatically follow MediaWiki by default, but with a way to override. In addition, for a major switchover we use orchestration scripts. In those scripts, MediaWiki is merely one among many, each of which is semi-automatically switched together.

This isn't documented, and at the time I reviewed navtiming's etcd code, it made sense as-is, but in retrospect, we probably shouldn't read mediawiki-config/WMFMasterDatacenter from etcd, given no other service does (besides mediawiki).

added mapped IPv6 addresses on the interface for webperf1001/2001:

https://gerrit.wikimedia.org/r/#/c/433299/

after applying that and confirming we got the new IPs from puppet i added the records in DNS:

https://gerrit.wikimedia.org/r/#/c/434423/

I confirmed with host and ping6 connectivity between 1001/2001, 2002/1001 etc.

There was no server reboot or service restart. Apache was already listening on tcp6 :::80.

This was to ensure that all webperf machines are treated the same. 1002/2002 came with v6 from the start.

If there are any unexpected issue then reverting is reverting the 2 patches above. I used a TTL of 5 min in DNS just in case to minimize downtime as requested.

22:50 Krinkle: webperf1001: restart navtiming service, to test with new ipv6 capabilities

The service remained fine after restart. Depending on the target (kafka vs statsd) it'll have either started using ipv6 without issue, or keeps using ipv4 without issue. Kafka hosts have ipv6, but graphite/statsd not (I think).

The actual network interface and DNS were added about 45min earlier at 22:05, and looking back, right after Puppet enabled the net6 interface, an unexpected outage started for navtiming, which apparently lasted for 10min before it corrected itself.

The logs on webperf1001 don't show any messages relating to navtiming in either journal nor syslog. The service had an uptime of 3 weeks before I restarted it at 22:50.

The burst of data around 22:15 suggests that not all the data was lost during the 10min gap, but that some was somehow buffered or clogged somewhere, and then made its way from webperf1001 to statsd at 22:15. It'd be interesting to know what caused it to stay offline for so long (10min), and where (some of the) data was kept during that time, and whether we can do something to improve that within navtiming.py.

Krinkle mentioned this in T195312: Move flame graphs hosting from mwlog1001 to webperf-2 and enable in Beta Cluster.May 22 2018, 4:40 PM

Change 433710 abandoned by Imarlier:
webperf: separate permissions from specific apps

https://gerrit.wikimedia.org/r/433710

Change 433710 restored by Imarlier:
webperf: separate permissions from specific apps

https://gerrit.wikimedia.org/r/433710

In T158837#4221485, @Krinkle wrote:

@Imarlier I was just thinking about whether the etcd code is live or not for webperf/navtiming. The puppet invocation hasn't changed, but given navtiming.systemd uses the run_navtiming.py entry point (which reads config.ini), that presumably means the etcd_path/etcd_domain are used in prod. Is that right?

Anyhow, it currently uses MediaWiki's WMFMasterDatacenter key which seems inappropriate. It certainly seems desirable that, in the event of an overall switch-over that webperf/navtiming follows the other services, but we should also be able to switch over independently. Especially for navtiming, which would otherwise have downtime in the case of server maintenance.

I was discussing this with @Dzahn on IRC just now (who was asking whether it's fine to reboot the server to try enabling IPv6), which made me think it'd be nice if we could just switch over to codfw for a bit, upgrade the eqiad server, and then be able to switch back to try out, and switch back again if it doesn't work work. Instead of being subject to downtime.

Puppet has various inheritance options we can use to make our etcd config automatically follow MediaWiki by default, but with a way to override. In addition, for a major switchover we use orchestration scripts. In those scripts, MediaWiki is merely one among many, each of which is semi-automatically switched together.

This isn't documented, and at the time I reviewed navtiming's etcd code, it made sense as-is, but in retrospect, we probably shouldn't read mediawiki-config/WMFMasterDatacenter from etcd, given no other service does (besides mediawiki).

@Krinkle I'm fine with creating a different etcd key, as long as there's an easy way to change that key if/when a full data center failover happens. Sounds like that just means a modification to an orchestration script, which is good enough for me.

What I do _not_ want to do is to add anything to puppet in order to do this :-)

Change 433710 merged by Dzahn:
[operations/puppet@production] webperf: Make the different webperf roles explicit

https://gerrit.wikimedia.org/r/433710

Krinkle updated the task description. (Show Details)Jun 4 2018, 9:04 PM

Change 439648 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] Need to install mongodb on xhgui machines

https://gerrit.wikimedia.org/r/439648

• Imarlier closed subtask T196406: Decom "xhprof" viewer as Resolved.Jun 12 2018, 4:13 PM

Change 439648 merged by Herron:
[operations/puppet@production] Need to install mongodb on xhgui machines

https://gerrit.wikimedia.org/r/439648

Change 440649 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[performance/docroot@master] Make links from php-profiling.html to to /xenon/ relative

https://gerrit.wikimedia.org/r/440649

Change 440649 merged by jenkins-bot:
[performance/docroot@master] Make links from php-profiling.html to to /xenon/ relative

https://gerrit.wikimedia.org/r/440649

Krinkle added subtasks: T195314: Set up webperf-1 node in Beta Cluster, T195312: Move flame graphs hosting from mwlog1001 to webperf-2 and enable in Beta Cluster.Jun 26 2018, 10:14 PM

Krinkle updated the task description. (Show Details)Jun 26 2018, 10:32 PM

Note to self with regards to the perf-site. The site's Apache configuration (src) currently hardcodes ServerName performance.wikimedia.org as well as the proxy backends for XHGui (tungsten) and Xenon data (mwlog1001).

We should make these parameters of the conf.erb template so that they can point to something else in Beta Cluster. Possibly made optional as well.

• Vvjjkkii reopened subtask T196406: Decom "xhprof" viewer as Open.Jul 1 2018, 1:06 AM

CommunityTechBot closed subtask T196406: Decom "xhprof" viewer as Resolved.Jul 2 2018, 4:41 AM

Krinkle closed subtask T195314: Set up webperf-1 node in Beta Cluster as Resolved.Jul 3 2018, 10:55 PM

Change 443752 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] webperf: Rename webperf profiles for clarity

https://gerrit.wikimedia.org/r/443752

Krinkle added a subtask: T180761: Move XHGui from tungsten to xhgui-001.Jul 17 2018, 1:36 AM

Krinkle added a subtask: T199853: Increase webperf1002/webperf2002 space from 50GB to 150GB (Ganeti).Jul 18 2018, 2:47 AM

Krinkle updated the task description. (Show Details)Jul 18 2018, 4:15 AM

Krinkle updated the task description. (Show Details)

Krinkle updated the task description. (Show Details)Jul 20 2018, 9:49 PM

akosiaris closed subtask T199853: Increase webperf1002/webperf2002 space from 50GB to 150GB (Ganeti) as Resolved.Jul 27 2018, 6:55 AM

Change 443752 merged by Giuseppe Lavagetto:
[operations/puppet@production] webperf: Rename webperf profiles for clarity

https://gerrit.wikimedia.org/r/443752

Change 452689 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] webperf: Add 'Server: <fqdn>' header to performance.wikimedia.org

https://gerrit.wikimedia.org/r/452689

Krinkle closed subtask T195312: Move flame graphs hosting from mwlog1001 to webperf-2 and enable in Beta Cluster as Resolved.Aug 14 2018, 3:51 PM

Krinkle updated the task description. (Show Details)Aug 14 2018, 3:55 PM

Krinkle updated the task description. (Show Details)

Change 452689 abandoned by Krinkle:
webperf: Add 'Server: <fqdn>' header to performance.wikimedia.org

Reason:
Might reconsider at a later point.

https://gerrit.wikimedia.org/r/452689

Krinkle reopened subtask T196406: Decom "xhprof" viewer as Open.Dec 21 2018, 3:30 AM

Krinkle mentioned this in T196406: Decom "xhprof" viewer .Dec 22 2018, 12:34 AM

Krinkle closed subtask T196406: Decom "xhprof" viewer as Resolved.Jan 16 2019, 11:11 PM

Krinkle claimed this task.Jan 22 2019, 9:03 PM

Krinkle moved this task from Doing (old) to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.

(un-assigning the tracking task to reduce clutter on the workboard, see open sub tasks for current assignees)

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:05 PM

Change 503675 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] webperf: Remove arclamp subscriber from mwlog servers

https://gerrit.wikimedia.org/r/503675

Change 503675 merged by Effie Mouzeli:
[operations/puppet@production] webperf: Remove arclamp subscriber from mwlog servers

https://gerrit.wikimedia.org/r/503675

@Krinkle I have stopped and disabled xenon-log, excimer-log, and apache on mwlog* servers, and I have removed the arclamp-generate-svgs cron job. Let me know if there is anything else related to change 503675

Krinkle removed a project: Patch-For-Review.Jul 18 2019, 1:56 AM

Krinkle mentioned this in T189629: rename role::xenon.Aug 5 2019, 5:56 PM

Change 530771 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] hieradata: Move beta 'cache::app_directors' from Horizon to Puppet

https://gerrit.wikimedia.org/r/530771

gerritbot added a project: Patch-For-Review.Aug 18 2019, 12:55 PM

Change 530773 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] hieradata: Add 'performance.wikimedia.beta.wmflabs.org' routing

https://gerrit.wikimedia.org/r/530773

Dzahn updated the task description. (Show Details)Nov 20 2019, 6:45 PM

Dzahn added a subtask: T238098: vm request for xhgui.

Dzahn mentioned this in T238788: make xhgui::app role support stretch/buster and deploy on new xhgui machines.Nov 20 2019, 8:08 PM

Change 552357 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] webperf: switch xhgui_host from tungsten to xhgui1001

https://gerrit.wikimedia.org/r/552357

xhgui1001/2001 should now be about ready to replace tungsten. see T238788#5683132

Change 552362 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] xhgui: rsync mongodb data from tungsten to xhgui1001

https://gerrit.wikimedia.org/r/552362

Change 552362 merged by Dzahn:
[operations/puppet@production] xhgui: rsync mongodb data from tungsten to xhgui1001

https://gerrit.wikimedia.org/r/552362

Mentioned in SAL (#wikimedia-operations) [2019-11-22T00:46:45Z] <mutante> xhgui1001/xhgui2001 - rsyncing /srv/mongod from tungsten to /srv/tungsten/mongod/ on both new machines (T158837)

• Gilles assigned this task to • dpifke.Jan 7 2020, 11:41 AM

Krinkle removed a project: Patch-For-Review.Mar 27 2020, 4:57 PM

Change 530771 abandoned by Krinkle:
hieradata: Move beta 'cache::app_directors' from Horizon to Puppet

Reason:
This has since been done as part of the ATS migration, the backend mapping for those is now on puppet.git both for beta cluster and for prod.

https://gerrit.wikimedia.org/r/530771

Change 530773 merged by RLazarus:
[operations/puppet@production] hieradata: Include cache-text in Beta Cluster 'cache_hosts'

https://gerrit.wikimedia.org/r/530773

Krinkle moved this task from To-do: Goals, prioritized next 4 Quarters to To-do: Goals prioritized current Quarter on the Performance-Team board.Jul 17 2020, 12:07 AM

Change 552357 merged by Dzahn:
[operations/puppet@production] webperf: switch xhgui_host from tungsten to xhgui1001

https://gerrit.wikimedia.org/r/552357

Mentioned in SAL (#wikimedia-operations) [2020-08-13T22:03:41Z] <mutante> switching xhgui from tungsten to xhgui1001 - ran puppet on webperf*001 - T180761 T158837

Change 620130 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] delete role::xhgui::app

https://gerrit.wikimedia.org/r/620130

gerritbot added a project: Patch-For-Review.Aug 13 2020, 10:37 PM

Krinkle closed subtask T180761: Move XHGui from tungsten to xhgui-001 as Resolved.Aug 20 2020, 10:09 PM

Krinkle added a subtask: T260395: decom tungsten.

Krinkle updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2020-08-20T22:20:00Z] <mutante> permanently shut down tungsten.eqiad.wmnet T260395 T158837 T180761 T224549

Stashbot mentioned this in T224549: Track remaining jessie systems in production.Aug 20 2020, 10:20 PM

Stashbot mentioned this in T260395: decom tungsten.

Dzahn raised the priority of this task from Low to Medium.Aug 20 2020, 10:22 PM

Dzahn updated the task description. (Show Details)

Change 620130 merged by Dzahn:
[operations/puppet@production] delete role::xhgui::app

https://gerrit.wikimedia.org/r/620130

Change 621606 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cumin: update xhgui alias to apply to new role name

https://gerrit.wikimedia.org/r/621606

Change 621606 merged by Dzahn:
[operations/puppet@production] cumin: update xhgui alias to apply to new role name

https://gerrit.wikimedia.org/r/621606

From my side this ticket looks done now.

Krinkle closed this task as Resolved.Aug 21 2020, 4:15 AM

fgiunchedi awarded a token.Aug 24 2020, 8:29 AM

fgiunchedi mentioned this in T261274: Figure out switchover steps for mwlog hosts.Aug 26 2020, 8:20 AM

• Cmjohnson closed subtask T260395: decom tungsten as Resolved.Sep 2 2020, 7:25 PM