Page MenuHomePhabricator
Create Task
Maniphest T261274

Figure out switchover steps for mwlog hosts
Open, MediumPublic
Actions

Description

At the moment we have a standby host for mwlog in codfw (mwlog2001) although I couldn't find anywhere how ready these hosts are for a switchover (and how to perform one)

mwlog1001 serves a bunch of purposes:

  1. udp2log receiver for mediawiki logs (including debug)
  2. hosts redis for xenon/arclamp (mediawiki writes profiles there, cfr T158837)
  3. serve mediawiki logs via rsync (unencrypted) to analytics and dumps hosts
  4. generic udp2log receiver for (scap|scholarships|iegreview) and mirrors this data to logstash (cfr T205856)

Related Objects

Event Timeline

fgiunchedi created this task.Aug 26 2020, 8:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2020, 8:20 AM
jijiki triaged this task as High priority.Aug 26 2020, 3:04 PM
jijiki added a project: serviceops.
jijiki added a subscriber: RLazarus.
jijiki subscribed.

@RLazarus I am setting priority to high as the switchover is scheduled for next week.

fgiunchedi lowered the priority of this task from High to Medium.Aug 26 2020, 3:11 PM

@jijiki thank you for the triaging, we'll be likely skipping mwlog hosts this time around though (i.e. leave in eqiad)

lmata moved this task from Inbox to Backlog on the observability board.Aug 27 2020, 2:15 PM
Krinkle mentioned this in T231025: LegacyHandler.php: PHP Warning: Host lookup failed [-10002]: Unknown error -10002.Feb 3 2021, 7:47 PM
Legoktm added a project: Datacenter-Switchover.Jun 26 2021, 12:21 AM
Legoktm moved this task from Backlog to Services to switch on the Datacenter-Switchover board.Jun 26 2021, 1:34 AM
lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM
lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM
lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:40 AM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:27 PM
fgiunchedi mentioned this in T327277: Move excimer/arclamp redis from mwlog to arclamp hosts.Jan 18 2023, 1:23 PM
lmata moved this task from Inbox to Prioritized on the Observability-Logging board.Jan 20 2023, 2:11 PM
Clement_Goubert subscribed.Feb 20 2023, 1:21 PM

@fgiunchedi Is this still relevant? Are there some specific steps to be taken for T327920: March 2023 Datacenter Switchover ?

fgiunchedi added a comment.Feb 20 2023, 2:28 PM
In T261274#8629518, @Clement_Goubert wrote:

@fgiunchedi Is this still relevant? Are there some specific steps to be taken for T327920: March 2023 Datacenter Switchover ?

Yes still relevant, no specific steps for the switchover in the sense that to the best of my knowledge we won't be switching mwlog (I might be wrong here!). Though for all intents and purposes folks can (per)use mwlog2002 during the switchover since logs are mirrored there anyways

Clement_Goubert added a comment.Feb 20 2023, 2:48 PM
In T261274#8629767, @fgiunchedi wrote:
In T261274#8629518, @Clement_Goubert wrote:

@fgiunchedi Is this still relevant? Are there some specific steps to be taken for T327920: March 2023 Datacenter Switchover ?

Yes still relevant, no specific steps for the switchover in the sense that to the best of my knowledge we won't be switching mwlog (I might be wrong here!). Though for all intents and purposes folks can (per)use mwlog2002 during the switchover since logs are mirrored there anyways

There is nothing touching mwlog in the procedure or in the cookbooks since it's not a service or behind discovery. Thanks for the confirmation.

andrea.denisse subscribed.Mar 30 2023, 11:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL