Page MenuHomePhabricator
Create Task
Maniphest T204907

Scap is checking canary servers in dormant instead of active-dc
Closed, ResolvedPublic
Actions

Description

Summary

scap canaries hosts are hardcoded in operations/puppet.git in a hiera configuration file.

  • The switch other process should have an action to update list of canaries
  • Puppet roles mediawiki::canary_appserver, mediawiki::appserver::canary_api are apparently legacy/useless.
  • From discussion with SRE : scap dsh groups/list of canaries should move to conftool

From the last scap sync-file log:

01:14:16 Finished Canaries Synced (duration: 00m 03s)
01:14:16 Executing check 'Check endpoints for mw1279.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1276.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1261.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1264.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mwdebug1002.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mwdebug1001.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1263.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1262.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1278.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1277.eqiad.wmnet'
01:14:16 Executing check 'Check endpoints for mw1265.eqiad.wmnet'
01:14:16 Check 'Check endpoints for mw1276.eqiad.wmnet' failed: /wiki/{title} (Main Page) is CRITICAL: Test Main Page returned the unexpected status 503 (expecting: 200); /wiki/{title} (Special Version) is CRITICAL: Test Special Version returned the unexpected status 503 (expecting: 200); /w/api.php (Main Page pageprops) is CRITICAL: Test Main Page pageprops returned the unexpected status 503 (expecting: 200)

01:14:18 Finished Canary Endpoint Check Complete (duration: 00m 02s)
01:14:18 Waiting for canary traffic...
01:14:36 Executing check 'Logstash Error rate for mw1279.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1276.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1261.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1264.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mwdebug1002.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mwdebug1001.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1263.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1262.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1278.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1277.eqiad.wmnet'
01:14:36 Executing check 'Logstash Error rate for mw1265.eqiad.wmnet'
01:14:36 Finished sync-check-canaries (duration: 00m 23s)
01:14:36 Started sync-proxies

It should be checking canary servers in codfw instead because the eqiad ones are dormant / not useful.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+71 -40Use conftool to populate mw canaries in scap
operations/puppetproduction+11 -11Revert "scap: use mediawiki canaries from codfw"
operations/puppetproduction+36 -4scap: Move prefix from confd to key creation
operations/puppetproduction+15 -11scap: Replace an ugly hack with puppet 4 syntax
operations/puppetproduction+1 -1scap: Update logstash URL for mediawiki canaries
operations/puppetproduction+11 -11scap: use mediawiki canaries from codfw
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Sep 20 2018, 1:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2018, 1:20 AM
Krinkle added a project: Datacenter-Switchover.Sep 20 2018, 1:21 AM
gerritbot added a subscriber: gerritbot.Sep 20 2018, 1:56 PM

Change 461637 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] scap: use mediawiki canaries from codfw

https://gerrit.wikimedia.org/r/461637

gerritbot added a project: Patch-For-Review.Sep 20 2018, 1:56 PM
gerritbot added a comment.Sep 20 2018, 2:22 PM

Change 461637 merged by Alexandros Kosiaris:
[operations/puppet@production] scap: use mediawiki canaries from codfw

https://gerrit.wikimedia.org/r/461637

Stashbot added a subscriber: Stashbot.Sep 20 2018, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-20T14:28:33Z] <hashar@deploy1001> Synchronized typos: Dummy sync to verify list of canaries for T204907 (duration: 00m 59s)

hashar added a subscriber: hashar.Sep 20 2018, 2:29 PM

After discussion with Alexandros and Giuseppe, for now we have just updated the list of hosts in the dsh files. Now we have:

14:28:03 Finished Canaries Synced (duration: 00m 03s)
14:28:03 Executing check 'Check endpoints for mw2218.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2217.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2226.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2225.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mwdebug2002.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2216.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2215.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mwdebug2001.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2227.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2224.codfw.wmnet'
14:28:03 Executing check 'Check endpoints for mw2228.codfw.wmnet'
14:28:08 Finished Canary Endpoint Check Complete (duration: 00m 04s)
14:28:08 Waiting for canary traffic...
14:28:23 Executing check 'Logstash Error rate for mw2218.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2217.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2226.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2225.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mwdebug2002.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2216.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2215.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mwdebug2001.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2227.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2224.codfw.wmnet'
14:28:23 Executing check 'Logstash Error rate for mw2228.codfw.wmnet'
14:28:24 Finished sync-check-canaries (duration: 00m 24s)

A better fix would have to be figured out to update those files when the primary datacenter is switched. Either we can keep this task or fill another one.

Krinkle mentioned this in T121597: Implement MediaWiki pre-promote checks.Sep 20 2018, 8:12 PM
hashar mentioned this in T191068: 1.32.0-wmf.22 deployment blockers.Sep 20 2018, 9:15 PM
greg removed a parent task: T191068: 1.32.0-wmf.22 deployment blockers.Sep 25 2018, 1:39 AM
hashar updated the task description. (Show Details)Sep 25 2018, 10:21 AM
hashar edited projects, added SRE, Wikimedia-Incident; removed Patch-For-Review.
hashar moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
greg edited projects, added Release-Engineering-Team (Watching / External); removed Release-Engineering-Team.EditedSep 25 2018, 2:25 PM
greg added a subscriber: greg.

(I assume SRE will do the adding to conftool and the editing/extending of the relevant dc switchover cookbook.)

Krinkle added a comment.Sep 27 2018, 6:46 PM

In addition to the servers Scap checks, there is also the url it reports when finding it issue.

16:34:42 Executing check 'Logstash Error rate for mw2224.codfw.wmnet'
16:34:42 Executing check 'Logstash Error rate for mw2228.codfw.wmnet'
16:34:42 Check 'Logstash Error rate for mw2224.codfw.wmnet' failed: ERROR: 50% OVER_THRESHOLD (Avg. Error rate: Before: 0.09, After: 2.00, Threshold: 1.00)

16:34:42 Canary error check failed for 1 canaries, less than threshold to halt deployment (2/11), see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details. Continuing...
16:34:42 Finished sync-check-canaries (duration: 00m 24s)

It now checks Codfw servers, but the url https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 is a shortcut for eqiad canaries still. We should probably make this a regular dashboard that always shows both, without needing to vary it, or to use a query query parameter so that it can dynamically be set to the current set of canaries.

Krinkle added a comment.Sep 27 2018, 6:46 PM

(I see James reported that at T205559.)

akosiaris claimed this task.Sep 28 2018, 9:45 AM
akosiaris merged a task: T205559: Scap canary warning monitoring URL is hard-coded with eqiad servers, so isn't useful when codfw is primary.Sep 28 2018, 11:02 AM
akosiaris added a subscriber: Jdforrester-WMF.
akosiaris added a comment.Sep 28 2018, 11:09 AM

As far as solving the logstash URL I think the best approach would be to just have the entire list in a dashboard. I 've updated the "scap canary" dashboard. However for sharing purposes this generated a new URL https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040. I 'll update scap.cfg with it

gerritbot added a comment.Sep 28 2018, 11:11 AM

Change 463453 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] scap: Update logstash URL for mediawiki canaries

https://gerrit.wikimedia.org/r/463453

gerritbot added a project: Patch-For-Review.Sep 28 2018, 11:11 AM
gerritbot added a comment.Sep 28 2018, 11:24 AM

Change 463453 merged by Alexandros Kosiaris:
[operations/puppet@production] scap: Update logstash URL for mediawiki canaries

https://gerrit.wikimedia.org/r/463453

gerritbot added a comment.Sep 28 2018, 1:02 PM

Change 463469 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Use conftool to populate mw canaries in scap

https://gerrit.wikimedia.org/r/463469

hashar updated the task description. (Show Details)Sep 30 2018, 9:08 PM
gerritbot added a comment.Oct 1 2018, 7:52 AM

Change 463708 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] scap: Replace an ugly hack with puppet 4 syntax

https://gerrit.wikimedia.org/r/463708

gerritbot added a comment.Oct 1 2018, 7:52 AM

Change 463709 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] WIP: scap: Move prefix from confd to key creation

https://gerrit.wikimedia.org/r/463709

gerritbot added a comment.Oct 2 2018, 7:54 AM

Change 463708 merged by Alexandros Kosiaris:
[operations/puppet@production] scap: Replace an ugly hack with puppet 4 syntax

https://gerrit.wikimedia.org/r/463708

Joe mentioned this in D1117: Only check logstash for canaries in the active datacenter.Oct 2 2018, 1:06 PM
gerritbot added a comment.Oct 10 2018, 5:02 PM

Change 465661 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Revert "scap: use mediawiki canaries from codfw"

https://gerrit.wikimedia.org/r/465661

gerritbot added a comment.Oct 10 2018, 5:03 PM

Change 465661 merged by Alexandros Kosiaris:
[operations/puppet@production] Revert "scap: use mediawiki canaries from codfw"

https://gerrit.wikimedia.org/r/465661

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:24 PM
jijiki mentioned this in T218328: Scap2 to use etcd for target servers.Mar 15 2019, 11:42 AM
• Phabricator_maintenance added a project: Release-Engineering-Team-TODO.Jun 12 2019, 11:43 PM
• Phabricator_maintenance moved this task from Should be empty (use Release-Engineering-Team) to Watching/External on the Release-Engineering-Team-TODO board.Jun 12 2019, 11:48 PM
• Phabricator_maintenance removed a project: Release-Engineering-Team (Watching / External).Jun 12 2019, 11:48 PM
greg added a project: Release-Engineering-Team.Jun 21 2019, 10:34 PM
greg edited projects, added Release-Engineering-Team (Deployment services); removed Release-Engineering-Team.Jun 24 2019, 9:20 PM
gerritbot added a comment.Apr 1 2020, 10:12 AM

Change 463469 abandoned by Alexandros Kosiaris:
Use conftool to populate mw canaries in scap

Reason:
https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /465411/ was merged, let's abandon this

https://gerrit.wikimedia.org/r/463469

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Aklapper edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Oct 14 2020, 5:09 PM
akosiaris closed this task as Resolved.Oct 30 2020, 5:16 PM

This was done, resolving.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL