Page MenuHomePhabricator

deployment-cache instances are missing several major features available in production
Closed, ResolvedPublic

Description

deployment-cache instances should be recreated using HAProxy as the TLS terminator and ATS 9.x as the backend caching layer.

Event Timeline

Vgutierrez changed the task status from Open to Stalled.Oct 17 2022, 9:21 AM

Waiting on T320932

Vgutierrez changed the task status from Stalled to In Progress.Oct 17 2022, 10:48 AM

Spawning deployment-cache-text07 && deployment-cache-upload07...

Change 843470 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Allow disabling monitoring

https://gerrit.wikimedia.org/r/843470

Change 843470 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Allow disabling monitoring

https://gerrit.wikimedia.org/r/843470

Change 843491 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish::frontend: Allow disable monitoring

https://gerrit.wikimedia.org/r/843491

Change 843491 merged by Vgutierrez:

[operations/puppet@production] varnish::frontend: Allow disable monitoring

https://gerrit.wikimedia.org/r/843491

deployment-cache-text07 is up & running:

vgutierrez@deployment-cache-text07:~$ curl --connect-to en.wikipedia.beta.wmflabs.org:443:127.0.0.1 https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -I 
HTTP/2 200 
date: Mon, 17 Oct 2022 13:16:00 GMT
server: deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud
x-content-type-options: nosniff
content-language: en
content-security-policy: script-src 'unsafe-eval' blob: 'self' meta.wikimedia.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org 'unsafe-inline' login.wikimedia.beta.wmflabs.org; default-src 'self' data: blob: https://upload.wikimedia.beta.wmflabs.org upload.wikimedia.beta.wmflabs.org https://commons.wikimedia.beta.wmflabs.org https://upload.wikimedia.org https://commons.wikimedia.org meta.wikimedia.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org wikimedia.org en.wikisource.beta.wmflabs.org en.wikibooks.beta.wmflabs.org en.wikiversity.beta.wmflabs.org en.wikiquote.beta.wmflabs.org en.wikinews.beta.wmflabs.org en.wikivoyage.beta.wmflabs.org en.wiktionary.beta.wmflabs.org deployment.wikimedia.beta.wmflabs.org commons.wikimedia.beta.wmflabs.org api.wikimedia.beta.wmflabs.org login.wikimedia.beta.wmflabs.org; style-src 'self' data: blob: https://upload.wikimedia.beta.wmflabs.org upload.wikimedia.beta.wmflabs.org https://commons.wikimedia.beta.wmflabs.org https://upload.wikimedia.org https://commons.wikimedia.org meta.wikimedia.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org wikimedia.org 'unsafe-inline'; object-src 'none'; report-uri /w/api.php?action=cspreport&format=json
vary: Accept-Encoding,Cookie,Authorization
last-modified: Mon, 17 Oct 2022 11:50:33 GMT
content-type: text/html; charset=UTF-8
age: 64
x-cache: deployment-cache-text07 miss, deployment-cache-text07 hit/4
x-cache-status: hit-remote
server-timing: cache;desc="hit-remote", host;desc="deployment-cache-text07"
report-to: { "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}
set-cookie: WMF-Last-Access=17-Oct-2022;Path=/;HttpOnly;secure;Expires=Fri, 18 Nov 2022 12:00:00 GMT
set-cookie: WMF-Last-Access-Global=17-Oct-2022;Path=/;Domain=.wikipedia.beta.wmflabs.org;HttpOnly;secure;Expires=Fri, 18 Nov 2022 12:00:00 GMT
accept-ch: Sec-CH-UA-Arch,Sec-CH-UA-Bitness,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-UA-Platform-Version
permissions-policy: interest-cohort=(),ch-ua-arch=(self "intake-analytics.wikimedia.org"),ch-ua-bitness=(self "intake-analytics.wikimedia.org"),ch-ua-full-version-list=(self "intake-analytics.wikimedia.org"),ch-ua-model=(self "intake-analytics.wikimedia.org"),ch-ua-platform-version=(self "intake-analytics.wikimedia.org")
x-client-ip: 127.0.0.1
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: GeoIP=:::::v4; Path=/; secure; Domain=.beta.wmflabs.org
accept-ranges: bytes
content-length: 60787
vgutierrez@deployment-cache-text07:~$ sudo -i netstat --listen -n -p |grep 443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      711/haproxy         
tcp6       0      0 :::443                  :::*                    LISTEN      711/haproxy         
vgutierrez@deployment-cache-text07:~$ dpkg -l trafficserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-========================================================================
ii  trafficserver  9.1.3-1wm2   amd64        fast, scalable and extensible HTTP/1.1 and HTTP/2.0 caching proxy server

deployment-cache-upload07 is up & running as well:

vgutierrez@deployment-cache-upload07:~$ curl -I --connect-to upload.wikimedia.beta.wmflabs.org:443:127.0.0.1 https://upload.wikimedia.beta.wmflabs.org/wikipedia/commons/d/de/123_4.jpg
HTTP/2 200 
content-length: 200456
x-object-meta-sha1base36: l3g1j3ermq3z1om07dpn18k7wq6dtis
last-modified: Thu, 20 Jan 2022 19:42:40 GMT
etag: 0d1cc1db7e0c7e206cd365a39bd14e12
content-type: image/jpeg
date: Mon, 17 Oct 2022 13:37:06 GMT
server: ATS/9.1.3
age: 4
x-cache: deployment-cache-upload07 miss, deployment-cache-upload07 hit/1
x-cache-status: hit-remote
server-timing: cache;desc="hit-remote", host;desc="deployment-cache-upload07"
report-to: { "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}
accept-ch: Sec-CH-UA-Arch,Sec-CH-UA-Bitness,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-UA-Platform-Version
permissions-policy: interest-cohort=(),ch-ua-arch=(self "intake-analytics.wikimedia.org"),ch-ua-bitness=(self "intake-analytics.wikimedia.org"),ch-ua-full-version-list=(self "intake-analytics.wikimedia.org"),ch-ua-model=(self "intake-analytics.wikimedia.org"),ch-ua-platform-version=(self "intake-analytics.wikimedia.org")
x-client-ip: 127.0.0.1
access-control-allow-origin: *
access-control-expose-headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache
timing-allow-origin: *
accept-ranges: bytes
vgutierrez@deployment-cache-upload07:~$ dpkg -l trafficserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-========================================================================
ii  trafficserver  9.1.3-1wm2   amd64        fast, scalable and extensible HTTP/1.1 and HTTP/2.0 caching proxy server
vgutierrez@deployment-cache-upload07:~$ sudo -i netstat --listen -n -p |grep 443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      734/haproxy         
tcp6       0      0 :::443                  :::*                    LISTEN      734/haproxy

Mentioned in SAL (#wikimedia-releng) [2022-10-17T13:54:23Z] <vgutierrez> switch 185.15.56.35 from deployment-cache-upload06 to deployment-cache-upload07 - T320930

Mentioned in SAL (#wikimedia-releng) [2022-10-17T13:56:16Z] <vgutierrez> switch 185.15.56.36 from deployment-cache-text06 to deployment-cache-text07 - T320930

Change 843500 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hieradata::deployment-prep: Bump deployment-cache-text|upload instances

https://gerrit.wikimedia.org/r/843500

Mentioned in SAL (#wikimedia-releng) [2022-10-17T14:01:58Z] <vgutierrez> shutdown deployment-cache-(text|upload)06 - T320930

Change 843500 merged by Vgutierrez:

[operations/puppet@production] hieradata::deployment-prep: Bump deployment-cache-text|upload instances

https://gerrit.wikimedia.org/r/843500

Vgutierrez claimed this task.

Mentioned in SAL (#wikimedia-releng) [2022-10-18T08:03:36Z] <vgutierrez> wipe deployment-cache-(text|upload)06 - T320930