Page MenuHomePhabricator
Create Task
Maniphest T257840

fix planet.wm.org redirect nitpick (was: missing from planet.discovery.wmnet Subject Alternative Name)
Closed, ResolvedPublic
Actions

Description

I have observed the following error while looking at unrelated things in the logs of ats-be:

Jul 13 09:04:09 cp1081 traffic_manager[1133]: [Jul 13 09:04:09.249] [ET_NET 63] ERROR: SSL connection failed for 'planet.wikimedia.org': error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Indeed the public hostname "planet.wikimedia.org" is missing from the certificate subjectAltName:

DNS:planet.discovery.wmnet, DNS:planet.svc.eqiad.wmnet, DNS:planet.svc.codfw.wmnet, DNS:planet1001.eqiad.wmnet, DNS:planet2001.codfw.wmnet, DNS:*.planet.wikimedia.org

Note that *.planet.wm.org is there, but planet.wm.org is missing.

We should:

  • fix the certificate
  • make sure these kind of errors end up in logstash

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
planet: fix redirect to meta page, stop appending request URI pathDzahnoperations/puppetproduction+1 -1
ssl/planet: update cert for planet.discovery.wmnetDzahnoperations/puppetproduction+21 -23
ATS: send 'SSL connection failed' errors to logstashEmaoperations/puppetproduction+2 -1
Customize query in gerrit

Event Timeline

• ema created this task.Jul 13 2020, 2:18 PM
Restricted Application added a project: SRE. · View Herald TranscriptJul 13 2020, 2:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot added a comment.Jul 13 2020, 2:20 PM

Change 612282 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: send 'SSL connection failed' errors to logstash

https://gerrit.wikimedia.org/r/612282

gerritbot added a project: Patch-For-Review.Jul 13 2020, 2:20 PM
Dzahn claimed this task.Jul 13 2020, 9:51 PM
Dzahn added a comment.Jul 13 2020, 9:53 PM

It was never expected that planet without a language prefix exists.

It's just that i recently added the "blank" domain to config because i wanted to fix that it shows the error page if somebody manually enters it, maybe trying to find the overview of all existing languages.

And I did not finish that yet.. what you describe is the next step, add it to the cert. I'll do that. But it's not like something here was broken that wasn't like this for a long time.

• ema added a comment.Jul 14 2020, 2:59 PM
In T257840#6302796, @Dzahn wrote:

And I did not finish that yet.. what you describe is the next step, add it to the cert. I'll do that. But it's not like something here was broken that wasn't like this for a long time.

Yes, thanks @Dzahn! I spotted this in the logs only now. :)

• ema triaged this task as Low priority.Jul 14 2020, 2:59 PM
gerritbot added a comment.Jul 15 2020, 9:25 AM

Change 612282 merged by Ema:
[operations/puppet@production] ATS: send 'SSL connection failed' errors to logstash

https://gerrit.wikimedia.org/r/612282

• ema updated the task description. (Show Details)Jul 15 2020, 9:26 AM
• ema moved this task from Backlog to Scheduled work for this quarter on the Traffic board.Jul 15 2020, 9:50 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 15 2020, 10:10 AM
Stashbot added a comment.Jul 15 2020, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-15T17:32:37Z] <mutante> puppetmaster - revoking cert for planet.discovery.wmnet, add planet.wikimedia.org, remove planet.svc records, remove specific and outdated hostnames (T257840)

gerritbot added a comment.Jul 15 2020, 5:42 PM

Change 612904 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl/planet: update cert for planet.discovery.wmnet

https://gerrit.wikimedia.org/r/612904

gerritbot added a project: Patch-For-Review.Jul 15 2020, 5:42 PM
gerritbot added a comment.Jul 15 2020, 5:45 PM

Change 612904 merged by Dzahn:
[operations/puppet@production] ssl/planet: update cert for planet.discovery.wmnet

https://gerrit.wikimedia.org/r/612904

Dzahn raised the priority of this task from Low to Medium.Jul 15 2020, 5:48 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 15 2020, 6:10 PM
Dzahn updated the task description. (Show Details)Jul 15 2020, 6:16 PM
Dzahn closed this task as Resolved.Jul 15 2020, 6:20 PM

@ema Cert has been fixed. I added planet.wikimedia.org in addition to *.planet.wikimedia.org and removed "svc.eqiad/codfw" records and hardcoded host names that did not exist anymore (1001/2001).

before: DNS:planet.discovery.wmnet, DNS:planet.svc.eqiad.wmnet, DNS:planet.svc.codfw.wmnet, DNS:planet1001.eqiad.wmnet, DNS:planet2001.codfw.wmnet, DNS:*.planet.wikimedia.org

now: DNS:planet.discovery.wmnet, DNS:planet.wikimedia.org, DNS:*.planet.wikimedia.org

With existing Apache config https://planet.wikimedia.org redirects to https://meta.wikimedia.org/wiki/Planet_Wikimedia

You should not see these errors anymore.

• ema added a comment.Jul 16 2020, 8:36 AM
In T257840#6309655, @Dzahn wrote:

With existing Apache config https://planet.wikimedia.org redirects to https://meta.wikimedia.org/wiki/Planet_Wikimedia

That works! Nitpick: https://planet.wikimedia.org/test should probably send me to https://meta.wikimedia.org/wiki/Planet_Wikimedia, instead it points to https://meta.wikimedia.org/wiki/Planet_Wikimediatest

You should not see these errors anymore.

Indeed, thank you.

Dzahn renamed this task from planet.wm.org missing from planet.discovery.wmnet Subject Alternative Name to fix planet.wm.org redirect nitpick (was: missing from planet.discovery.wmnet Subject Alternative Name).Jul 27 2020, 8:45 PM
Dzahn reopened this task as Open.
Dzahn lowered the priority of this task from Medium to Low.
gerritbot added a comment.Aug 21 2020, 7:55 PM

Change 621802 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: fix redirect to meta page, stop appending request URI path

https://gerrit.wikimedia.org/r/621802

gerritbot added a project: Patch-For-Review.Aug 21 2020, 7:55 PM
gerritbot added a comment.Aug 21 2020, 8:01 PM

Change 621802 merged by Dzahn:
[operations/puppet@production] planet: fix redirect to meta page, stop appending request URI path

https://gerrit.wikimedia.org/r/621802

Dzahn closed this task as Resolved.Aug 21 2020, 8:04 PM
In T257840#6311364, @ema wrote:

Nitpick: https://planet.wikimedia.org/test should probably send me to https://meta.wikimedia.org/wiki/Planet_Wikimedia, instead it points to https://meta.wikimedia.org/wiki/Planet_Wikimediatest

Fixed by using RedirectMatch ^

curl -vvv https://planet.wikimedia.org/test
< location: https://meta.wikimedia.org/wiki/Planet_Wikimedia

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits