Page MenuHomePhabricator

Move oxygen to weblog1001
Closed, ResolvedPublic

Description

In T207760 weblog1001 was set up to replace oxygen. This task tracks the work to move from one host to the other.

Event Timeline

elukey triaged this task as Medium priority.Dec 13 2018, 2:13 PM
elukey created this task.

Change 479443 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafkatee::webrequest::ops: move hiera to parameters

https://gerrit.wikimedia.org/r/479443

Change 479443 merged by Elukey:
[operations/puppet@production] profile::kafkatee::webrequest::ops: move hiera to parameters

https://gerrit.wikimedia.org/r/479443

Change 479444 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add IPv6 mapped config to weblog1001

https://gerrit.wikimedia.org/r/479444

Change 479444 merged by Elukey:
[operations/puppet@production] Add IPv6 mapped config to weblog1001

https://gerrit.wikimedia.org/r/479444

Change 479447 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add AAAA/PTR records for weblog1001

https://gerrit.wikimedia.org/r/479447

Change 479447 merged by Elukey:
[operations/dns@master] Add AAAA/PTR records for weblog1001

https://gerrit.wikimedia.org/r/479447

Change 479448 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Swap oxygen with weblog1001

https://gerrit.wikimedia.org/r/479448

Change 479448 merged by Elukey:
[operations/puppet@production] Swap oxygen with weblog1001

https://gerrit.wikimedia.org/r/479448

Current status:

  • disabled puppet on oxygen and killed the kafkatee output to logstash (verified via netstat that we don't have any conn to logstash anymore)
  • enabled role logging::webrequest::ops on weblog1001, everything seems running fine

On Monday I will complete the work doing the following:

  • rsync/copy the archive dir from oxygen to weblog1001 (after logrotation of a couple of days we shouldn't have any "hole" in data)
  • apply role spare::system to oxygen and start the decom process.

@herron decided to proceed to unblock the oxygen's decom process, from now on we can decide how to proceed with logstash/webrequest-503 (it will likely take a bit of time so better to nuke oxygen in the meantime :). Hope that it is ok!

Change 480012 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafkatee::webrequest::ops: ensure archive directory

https://gerrit.wikimedia.org/r/480012

Change 480012 merged by Elukey:
[operations/puppet@production] profile::kafkatee::webrequest::ops: ensure archive directory

https://gerrit.wikimedia.org/r/480012

Change 480025 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafkatee::webrequest::ops: add quickdatacopy

https://gerrit.wikimedia.org/r/480025

Change 480025 merged by Elukey:
[operations/puppet@production] profile::kafkatee::webrequest::ops: add quickdatacopy

https://gerrit.wikimedia.org/r/480025

Change 480028 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Temporary assigning role::logging::webrequest::ops to oxygen

https://gerrit.wikimedia.org/r/480028

Change 480028 merged by Elukey:
[operations/puppet@production] Temporary assigning role::logging::webrequest::ops to oxygen

https://gerrit.wikimedia.org/r/480028

Change 480033 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafkatee::webrequest::ops: widen allowed data via rsync

https://gerrit.wikimedia.org/r/480033

Change 480033 merged by Elukey:
[operations/puppet@production] profile::kafkatee::webrequest::ops: widen allowed data via rsync

https://gerrit.wikimedia.org/r/480033

Change 480422 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafkatee::webrequest::ops: absent quickdatacopy

https://gerrit.wikimedia.org/r/480422

Change 480423 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set role(spare::system) for oxygen

https://gerrit.wikimedia.org/r/480423

Change 480424 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::kafkatee::webrequest::ops: remove unused code

https://gerrit.wikimedia.org/r/480424

Change 480422 merged by Elukey:
[operations/puppet@production] profile::kafkatee::webrequest::ops: absent quickdatacopy

https://gerrit.wikimedia.org/r/480422

Change 480423 merged by Elukey:
[operations/puppet@production] Set role(spare::system) for oxygen

https://gerrit.wikimedia.org/r/480423

Change 480424 merged by Elukey:
[operations/puppet@production] profile::kafkatee::webrequest::ops: remove unused code

https://gerrit.wikimedia.org/r/480424

elukey claimed this task.

Move completed, we are ready to decom oxygen :)