Fix scap canary checks on beta cluster (Check failed: returned 302, expecting: 200)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Mar 29 2021, 10:56 AM

Description

At https://integration.wikimedia.org/ci/job/beta-scap-sync-world/

Scap canary checks always fail on the beta cluster:

13:30:48 10:30:48 Check 'Check endpoints for deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud' failed: /wiki/{title} (Main Page) is CRITICAL: Test Main Page returned the unexpected status 302 (expecting: 200); /wiki/{title} (Special Version) is CRITICAL: Test Special Version returned the unexpected status 302 (expecting: 200)

That's a redirect from http to https, mediawiki_canary_swagger_url uses http in wmcs and https in production.

Details

	Subject	Repo	Branch	Lines +/-
	scap: Use https for beta swagger checks	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	taavi	T278686 Fix scap canary checks on beta cluster (Check failed: returned 302, expecting: 200)
Open	None	T206158 Investigate setting up HTTPS directly on beta appservers
Resolved	jbond	T282930 cfssl: Support wildcard certs with envoy

Event Timeline

taavi created this task.Mar 29 2021, 10:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 29 2021, 10:56 AM

taavi added a subtask: T206158: Investigate setting up HTTPS directly on beta appservers.Mar 30 2021, 5:56 AM

Krinkle renamed this task from Fix scap canary checks on beta cluster to Fix scap canary checks on beta cluster (Check failed: returned 302, expecting: 200).May 14 2021, 9:00 PM

Krinkle updated the task description. (Show Details)

Krinkle subscribed.

For other people's context (I believe Majavah knows this already) - the reason this has been left broken for so long is that neither HTTP nor HTTPS works from Scap's point of view.

When scap targets localhost with an HTTP url, you get a connection but none of the expected HTTP 200 responses (MW will redirect to HTTPS in beta, just like it does in prod).

When scap targets localhost with an HTTPS url, it would work, if not for the fact that we never get to MW because the beta appservers don't yet support valid TLS connections.

The servers got cfssl certificates in the subtask, but guess they need alternative names too for the wikis?

taavi@deployment-deploy01:~$ service-checker-swagger deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud https://en.wikipedia.beta.wmflabs.org -s /spec.yaml
[...]
ssl.CertificateError: hostname 'en.wikipedia.beta.wmflabs.org' doesn't match 'appservers.svc.deployment-prep.eqiad1.wikimedia.cloud'
[...]

I tried adding them (efecf3a6ff78) but wildcards are somehow not yet functional, filed T282930 for that.

taavi claimed this task.May 17 2021, 1:05 PM

Restricted Application added a project: User-Majavah. · View Herald TranscriptMay 17 2021, 1:05 PM

Change 692318 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] scap: Use https for beta swagger checks

https://gerrit.wikimedia.org/r/692318

gerritbot added a project: Patch-For-Review.May 17 2021, 1:05 PM

Change 692318 merged by Jbond:

[operations/puppet@production] scap: Use https for beta swagger checks

https://gerrit.wikimedia.org/r/692318

Maintenance_bot removed a project: Patch-For-Review.May 17 2021, 2:11 PM

taavi closed this task as Resolved.May 17 2021, 2:20 PM