Can't load flame or coal graphs on performance.wikimedia.org (HTTP 502)
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	ori
	Oct 21 2019, 8:54 PM

Description

Steps to Reproduce:

Try to load https://performance.wikimedia.org/php-profiling/.

Actual Results:

$ curl -o/dev/null -Dheaders.txt https://performance.wikimedia.org/php-profiling/

HTTP/2 502
date: Mon, 21 Oct 2019 20:53:00 GMT
content-type: text/html
server: ATS/8.0.5
cache-control: no-store
content-language: en
x-ats-timestamp: 1571691180
vary: Accept-Encoding
x-varnish: 858825109
age: 0
x-cache: cp1075 miss, cp1085 miss
x-cache-status: miss
server-timing: cache;desc="miss"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=21-Oct-2019;Path=/;HttpOnly;secure;Expires=Fri, 22 Nov 2019 12:00:00 GMT
x-analytics: https=1;nocookies=1
x-client-ip: 2620:0:1003:1003:90a1:c975:158f:9c2d

Expected Results: Flame graphs.

Details

	Subject	Repo	Branch	Lines +/-
	ssl: re-issue cert for performance.discovery.wmnet	operations/puppet	production	+24 -23

Customize query in gerrit

Related Objects

Mentioned In: T236125: Trigger envoy reload upon TLS certificate update

Event Timeline

ori created this task.Oct 21 2019, 8:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2019, 8:54 PM

https://performance.wikimedia.org/php-profiling/

200 OK
age: 45
cache-control: s-maxage=600, max-age=600, public, must-revalidate
x-cache: cp1083 pass, cp3033 hit/1, cp3043 hit/1

https://performance.wikimedia.org/php-profiling/?bla

200 OK
cache-control: s-maxage=600, max-age=600, public, must-revalidate
x-cache: cp1079 pass, cp3043 miss, cp3043 pass

Krinkle added a project: Traffic.Oct 21 2019, 9:02 PM

Restricted Application added a project: SRE. · View Herald TranscriptOct 21 2019, 9:02 PM

Still doesn't work for me.

colewhite triaged this task as Medium priority.Oct 21 2019, 11:09 PM

Confirmed on WMCS:

HTTP/2 502 
date: Tue, 22 Oct 2019 07:34:28 GMT
content-type: text/html
server: ATS/8.0.5
cache-control: no-store
content-language: en
x-ats-timestamp: 1571729668
content-encoding: gzip
vary: Accept-Encoding
x-varnish: 735937566
age: 0
x-cache: cp1075 miss, cp1089 miss
x-cache-status: miss
server-timing: cache;desc="miss"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=22-Oct-2019;Path=/;HttpOnly;secure;Expires=Sat, 23 Nov 2019 00:00:00 GMT
x-analytics: https=1;nocookies=1
x-client-ip: 172.16.0.19

Compared to hitting esams:

HTTP/2 200 
date: Tue, 22 Oct 2019 07:33:51 GMT
content-type: text/html
server: Apache
last-modified: Sat, 06 Apr 2019 11:41:38 GMT
vary: Accept-Encoding
backend-timing: D=137 t=1571729631931853
cache-control: s-maxage=600, max-age=600, public, must-revalidate
etag: W/"2ce8-585db161b064a"
x-varnish: 241173347, 565242169, 124454678
accept-ranges: bytes
age: 0
x-cache: cp1083 pass, cp3043 miss, cp3040 pass
x-cache-status: miss
server-timing: cache;desc="miss"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=22-Oct-2019;Path=/;HttpOnly;secure;Expires=Sat, 23 Nov 2019 00:00:00 GMT
x-analytics: https=1;nocookies=1
x-client-ip: 185.228.228.207

Seems related to ATS.

The certificate for performance.discovery.wmnet does not include performance.wikimedia.org in SubjectAltName, hence ATS fails to connect to the origin via https. I'm working on a fix.

Change 545203 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ssl: re-issue cert for performance.discovery.wmnet

https://gerrit.wikimedia.org/r/545203

gerritbot added a project: Patch-For-Review.Oct 22 2019, 7:48 AM

Change 545203 merged by Ema:
[operations/puppet@production] ssl: re-issue cert for performance.discovery.wmnet

https://gerrit.wikimedia.org/r/545203

• ema mentioned this in T236125: Trigger envoy reload upon TLS certificate update.Oct 22 2019, 8:03 AM