Page MenuHomePhabricator

Move kartotherian/tilerator logging to new logging pipeline
Open, MediumPublic

Description

We need to move Maps (Kartotherian) and Maps (Tilerator) to the new logging pipeline. The config should be updated similar to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/change-propagation/deploy/+/500813 and newest node dependencies should be used (newest version of service-runner).

Event Timeline

MSantos created this task.May 2 2019, 4:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2019, 4:25 PM
MSantos triaged this task as Medium priority.May 2 2019, 4:28 PM
Pchelolo moved this task from Backlog to watching on the Services board.May 2 2019, 5:59 PM
Pchelolo edited projects, added Services (watching); removed Services.

Reminder/ping as we (SRE Observability) would like to deprecate all non-kafka inputs by end of Q4 FY19/20. If the service is moving (or has moved) to k8s then what's left to do is disable gelf log output and keep on stdout/stderr. If the service isn't moving to k8s then we'll also need to perform puppet-level changes. Thanks!

Change 602460 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[maps/kartotherian/deploy@master] Use new logging pipeline

https://gerrit.wikimedia.org/r/602460

Change 602461 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[maps/tilerator/deploy@master] Use new logging pipeline

https://gerrit.wikimedia.org/r/602461

Hi @fgiunchedi, thanks for the reviews! What's the minimum version of service-runner that's required for this? We may need to do a deployment for one or both services to bump dependency versions, but I'm not sure.

Hi @fgiunchedi, thanks for the reviews! What's the minimum version of service-runner that's required for this? We may need to do a deployment for one or both services to bump dependency versions, but I'm not sure.

I _think_ service-runner >= 2.6.18 or 2.6.19 will do, judging from https://phabricator.wikimedia.org/T211125#5206793 and related changes

Great, thanks @fgiunchedi! It looks like we currently have service-runner@2.7.3 in production for both services.

Change 602704 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/puppet@production] Include ::profile::rsyslog::udp_localhost_compat in OSM common role

https://gerrit.wikimedia.org/r/602704

Change 602704 merged by Filippo Giunchedi:
[operations/puppet@production] maps: profile::rsyslog::udp_localhost_compat

https://gerrit.wikimedia.org/r/602704

Change 602461 merged by jenkins-bot:
[maps/tilerator/deploy@master] Use new logging pipeline

https://gerrit.wikimedia.org/r/602461

Change 602460 merged by jenkins-bot:
[maps/kartotherian/deploy@master] Use new logging pipeline

https://gerrit.wikimedia.org/r/602460

Hi all, it looks like we've moved to syslog logging in https://gerrit.wikimedia.org/r/c/maps/kartotherian/deploy/+/602460 however the patch isn't deployed yet AFAICS?

Hi all, it looks like we've moved to syslog logging in https://gerrit.wikimedia.org/r/c/maps/kartotherian/deploy/+/602460 however the patch isn't deployed yet AFAICS?

You're right. However the changes are now being prepared for testing in the beta cluster and should land production afterwards, https://gerrit.wikimedia.org/r/c/maps/kartotherian/deploy/+/627503

lmata moved this task from Externally blocked to Radar on the observability board.Sep 21 2020, 8:28 PM
Mholloway removed Mholloway as the assignee of this task.Sep 30 2020, 6:21 PM
MSantos claimed this task.Oct 1 2020, 10:23 AM

@fgiunchedi we've been working on fixing OSM replication recently in the eqiad cluster, so we blocked deployments for safety. It's almost finished and if possible, we will try to deploy still this week or in the next. Thanks for your patience.

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:12:57Z] <mbsantos@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:24:33Z] <mbsantos@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932) (duration: 11m 36s)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:26:22Z] <mbsantos@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932) (duration: 01m 09s)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T21:11:45Z] <mbsantos@deploy1001> Started deploy [tilerator/deploy@97575e4]: Add new target for beta environment and clean-up old envs (T222377)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T21:14:08Z] <mbsantos@deploy1001> Finished deploy [tilerator/deploy@97575e4]: Add new target for beta environment and clean-up old envs (T222377) (duration: 02m 23s)