Page MenuHomePhabricator

Migrate logging::webrequest::ops to Buster
Closed, ResolvedPublic

Description

As per title, all hosts running this role should be on Buster. Ideally by FQ2 FY20-21.

root@cumin1001:~# cumin 'P{O:logging::webrequest::ops} and not P{F:lsbdistcodename = buster}'
1 hosts will be targeted:
weblog1001.eqiad.wmnet
DRY-RUN mode enabled, aborting

Event Timeline

herron added a subscriber: herron.Mar 23 2020, 3:34 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Apr 6 2020, 12:33 PM

The role works as-is in WMCS on a Buster image, I'll provision a new Buster VM in production

I misspoke, weblog1001 is a physical host and slated for decom >= 2021. I'm tempted to move it to a VM instead given the low resource usage. The downside I can think of is that when Ganeti is down we can't inspect webrequest, although the role is easy enough to apply it to e.g. centrallog in an emergency.

fgiunchedi triaged this task as Medium priority.Jul 9 2020, 8:57 AM
fgiunchedi added a project: User-fgiunchedi.

This won't really work for a Ganeti VM due to the size of /srv, currently 5T on mwlog1001. But Shirley has fixed Partman to retain /srv, so this could be used here? And for the time mwlog1001 is reimaged, data could flow to mwlog2001 for a few hours.

Also, something is off with your Cumin search from the task description, there's also mwlog2001 :-)

Scratch my previous comments, wrong role...

We might end up on physical hardware anyways after all, namely folding weblog into centrallog since they fulfill similar roles, the hardware is idle most of the time, and centrallog hosts are Buster already.

Change 610832 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] role: fold weblog into centrallog

https://gerrit.wikimedia.org/r/610832

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Mon, Jul 20, 12:48 PM

Change 610832 merged by Filippo Giunchedi:
[operations/puppet@production] role: fold weblog into centrallog

https://gerrit.wikimedia.org/r/610832

Change 615410 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] webrequest: add rsync server to migrate data

https://gerrit.wikimedia.org/r/615410

Change 615410 merged by Filippo Giunchedi:
[operations/puppet@production] webrequest: add rsync server to migrate data

https://gerrit.wikimedia.org/r/615410

Change 615420 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: sync weblog data to centrallog2001

https://gerrit.wikimedia.org/r/615420

Change 615420 merged by Filippo Giunchedi:
[operations/puppet@production] profile: sync weblog data to centrallog2001

https://gerrit.wikimedia.org/r/615420

Change 617388 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: ensure only one webrequest host sends 5xx to logstash

https://gerrit.wikimedia.org/r/617388

Change 617388 merged by Filippo Giunchedi:
[operations/puppet@production] profile: ensure only one webrequest host sends 5xx to logstash

https://gerrit.wikimedia.org/r/617388

fgiunchedi closed this task as Resolved.Fri, Jul 31, 9:07 AM
fgiunchedi claimed this task.

This is complete! weblog1001 decom is tracked at T259217: Decommission weblog1001 (unrack or return to spares)