Page MenuHomePhabricator
Create Task
Maniphest T185667

setup/install eventlog1002.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the setup and implementation of eventlog1002(WMF4751). Once online, it will replace eventlog1001, which will be decommissioned.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Fix eventlog1002's ipv6 addressoperations/dnsmaster+1 -1
fixing eventlog1002 entryoperations/dnsmaster+1 -1
setting dns entries for eventlog1002operations/dnsmaster+6 -0
eventlog1002 install paramsoperations/puppetproduction+10 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jan 24 2018, 10:30 PM
RobH triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2018, 10:30 PM
gerritbot subscribed.Jan 24 2018, 10:43 PM

Change 406127 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] setting dns entries for eventlog1002

https://gerrit.wikimedia.org/r/406127

gerritbot added a project: Patch-For-Review.Jan 24 2018, 10:43 PM
gerritbot added a comment.Jan 24 2018, 10:47 PM

Change 406127 merged by RobH:
[operations/dns@master] setting dns entries for eventlog1002

https://gerrit.wikimedia.org/r/406127

RobH updated the task description. (Show Details)Jan 24 2018, 10:47 PM
RobH mentioned this in T184551: EQIAD: (1) hardware request for eventlog1001 replacement - eventlog1002..
RobH updated the task description. (Show Details)Jan 24 2018, 10:56 PM
RobH reassigned this task from RobH to Ottomata.Jan 24 2018, 11:00 PM

@Ottomata: eventlog1001 is trusty. Can eventlog1002 be stretch or does it need to be an older distro? Please advise and assign back to me, thanks!

gerritbot added a comment.Jan 24 2018, 11:08 PM

Change 406129 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] eventlog1002 install params

https://gerrit.wikimedia.org/r/406129

gerritbot added a comment.Jan 24 2018, 11:11 PM

Change 406129 merged by RobH:
[operations/puppet@production] eventlog1002 install params

https://gerrit.wikimedia.org/r/406129

RobH removed a project: Patch-For-Review.Jan 24 2018, 11:11 PM
RobH updated the task description. (Show Details)
faidon renamed this task from setup/install evenlog1002.eqiad.wmnet to setup/install eventlog1002.eqiad.wmnet.Jan 25 2018, 6:30 PM
Ottomata updated the task description. (Show Details)Jan 25 2018, 11:15 PM
RobH claimed this task.Jan 26 2018, 5:48 PM
RobH updated the task description. (Show Details)
Ottomata added a comment.Jan 30 2018, 12:00 AM

@RobH, sorry I missed your ping on this, Trusty please! :)

faidon added a comment.Jan 30 2018, 12:26 AM

Trusty has about a year left of upstream support, and likely less for our own purposes. Any reason to not switch to somewhere more recent while we're at it?

Ottomata added a comment.Jan 30 2018, 12:39 AM

We are holding for Kubernetes! :) When it is ready, we will move the many individual processes (which are managed and monitored via a custom upstart based eventloggingctl scripts written by Ori) to k8s. We'd prefer not to have to do a bunch of work to change what we have now, just to do it again (hopefully next FY) to move to K8s.

faidon added a comment.Feb 1 2018, 10:41 AM

I had a look at both modules/eventlogging/files/eventloggingctl and modules/eventlogging/templates/upstart/*. They all seemed fairly easy to reimplement with systemd (with or without templates; for the former, a good reference would be e.g. the Tor package's units). It all feels to me like less than a day's effort unless I've gravely misunderstood how this all works and underestimating it.

IMHO, it would be better to do this effort now, rather than being pressured to do this Kubernetes migration in less than a year's time. Such an interdependency has the potential of being one of our last trusty strugglers, so I'd like to untangle it and get it out of the way, unless you're feeling strongly against this :)

MoritzMuehlenhoff subscribed.Feb 1 2018, 11:10 AM
Ottomata added a comment.Feb 1 2018, 4:08 PM

I actually tried to move to systemd a couple of years ago. I don’t
remember the exact details, but there were some serious difficulties in
automatically registering groups of processes to be managed together, even
with templates. Wildcards (*) sort of worked, but not in all cases.

Ottomata added a comment.Feb 1 2018, 4:22 PM

Ah, found previous ticket: T114199

MoritzMuehlenhoff added a comment.Feb 1 2018, 6:27 PM
In T185667#3937886, @Ottomata wrote:

I actually tried to move to systemd a couple of years ago.

But T114199 was for jessie, the first Debian release with systemd by default. With stretch a lot has happened since then (a full two years of activity of a fairly lively upstream project). We should really revisit this with stretch.

RobH updated the task description. (Show Details)Feb 1 2018, 6:53 PM
RobH added a comment.Feb 1 2018, 7:20 PM

Install blocked by network issue detailed on T186252 for onsite work.

RobH updated the task description. (Show Details)Feb 1 2018, 7:41 PM
gerritbot added a comment.Feb 2 2018, 5:00 PM

Change 407677 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] fixing eventlog1002 entry

https://gerrit.wikimedia.org/r/407677

gerritbot added a project: Patch-For-Review.Feb 2 2018, 5:00 PM
gerritbot added a comment.Feb 2 2018, 5:03 PM

Change 407677 merged by RobH:
[operations/dns@master] fixing eventlog1002 entry

https://gerrit.wikimedia.org/r/407677

• Cmjohnson closed subtask T185668: apply hostname labels to eventlog1001/WMF4751 as Resolved.Feb 2 2018, 5:29 PM
RobH removed a project: Patch-For-Review.Feb 2 2018, 5:30 PM
RobH updated the task description. (Show Details)
RobH reassigned this task from RobH to Ottomata.Feb 2 2018, 5:32 PM
RobH closed subtask T186252: check eventlog1002 production network cable as Resolved.
RobH added a subscriber: elukey.

So due to both Faidon and Mortiz's comments, I've gone ahead and installed with stretch. If it needs to be re-imaged to fall back to an older distro, then dhcp and reimage will have to be updated/run.

The initial puppet run is currently going, I'm escalating this to @elukey or @Ottomata (both have been involved in the setup of this host, so not sure who is ideal.)

RobH updated the task description. (Show Details)Feb 2 2018, 5:34 PM
elukey added a comment.Feb 2 2018, 5:35 PM
In T185667#3936830, @faidon wrote:

I had a look at both modules/eventlogging/files/eventloggingctl and modules/eventlogging/templates/upstart/*. They all seemed fairly easy to reimplement with systemd (with or without templates; for the former, a good reference would be e.g. the Tor package's units). It all feels to me like less than a day's effort unless I've gravely misunderstood how this all works and underestimating it.

We could to a research spike of one day and attempt to have something working in labs, and maybe discuss after it if it is worth to keep going with trusty/upstart or not?

• Nuria moved this task from Incoming to Wikistats on the Analytics board.Feb 5 2018, 5:33 PM
elukey added a comment.Feb 6 2018, 1:17 PM

All the work to move eventlogging to systemd is going to be tracked in https://phabricator.wikimedia.org/T114199, let's use this task only for the eventlog1002's productionization.

Milimetric moved this task from Wikistats to Kafka Work on the Analytics board.Mar 1 2018, 6:21 PM
Milimetric moved this task from Kafka Work to Operational Excellence on the Analytics board.
Ottomata merged a task: T182925: Hardware refresh eventlogging box.Mar 1 2018, 6:45 PM
elukey mentioned this in T188749: Deprecation of mw.errors.* metrics.Mar 2 2018, 4:36 PM
elukey added a comment.Mar 2 2018, 4:52 PM

As preparation step to make the migration I discovered that eventlog1001 seems to receive udp traffic from mwlog* hosts to parse and generate mw.errors.* metrics. I opened a task to remove all this code since it seems deprecated, but it might take a bit of time: https://phabricator.wikimedia.org/T188749

Also pinged folks in https://gerrit.wikimedia.org/r/#/c/415218/, it would be great not to create a ZMQ forwarder on eventlog1002.

elukey added a parent task: T114199: Upgrade eventlogging servers to Stretch.Mar 5 2018, 4:51 PM
ops-monitoring-bot subscribed.Mar 8 2018, 12:49 PM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['eventlog1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803081249_elukey_9444.log.

ops-monitoring-bot added a comment.Mar 8 2018, 1:20 PM

Completed auto-reimage of hosts:

['eventlog1002.eqiad.wmnet']

and were ALL successful.

elukey closed this task as Resolved.Mar 8 2018, 1:56 PM
elukey updated the task description. (Show Details)
gerritbot added a comment.Mar 11 2018, 8:11 AM

Change 418714 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Fix eventlog1002's ipv6 address

https://gerrit.wikimedia.org/r/418714

gerritbot added a project: Patch-For-Review.Mar 11 2018, 8:11 AM
gerritbot added a comment.Mar 12 2018, 8:47 AM

Change 418714 merged by Elukey:
[operations/dns@master] Fix eventlog1002's ipv6 address

https://gerrit.wikimedia.org/r/418714

elukey mentioned this in T196072: Analytics hosts missing in Inventory/Refresh.May 31 2018, 1:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits