deployment-cache instances should be recreated using HAProxy as the TLS terminator and ATS 9.x as the backend caching layer.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Vgutierrez | T320930 deployment-cache instances are missing several major features available in production | |||
Resolved | aborrero | T320932 Request increased quota for deployment-prep Cloud VPS project |
Event Timeline
Change 843470 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] cache::haproxy: Allow disabling monitoring
Change 843470 merged by Vgutierrez:
[operations/puppet@production] cache::haproxy: Allow disabling monitoring
Change 843491 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] varnish::frontend: Allow disable monitoring
Change 843491 merged by Vgutierrez:
[operations/puppet@production] varnish::frontend: Allow disable monitoring
deployment-cache-text07 is up & running:
vgutierrez@deployment-cache-text07:~$ curl --connect-to en.wikipedia.beta.wmflabs.org:443:127.0.0.1 https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -I HTTP/2 200 date: Mon, 17 Oct 2022 13:16:00 GMT server: deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud x-content-type-options: nosniff content-language: en content-security-policy: script-src 'unsafe-eval' blob: 'self' meta.wikimedia.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org 'unsafe-inline' login.wikimedia.beta.wmflabs.org; default-src 'self' data: blob: https://upload.wikimedia.beta.wmflabs.org upload.wikimedia.beta.wmflabs.org https://commons.wikimedia.beta.wmflabs.org https://upload.wikimedia.org https://commons.wikimedia.org meta.wikimedia.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org wikimedia.org en.wikisource.beta.wmflabs.org en.wikibooks.beta.wmflabs.org en.wikiversity.beta.wmflabs.org en.wikiquote.beta.wmflabs.org en.wikinews.beta.wmflabs.org en.wikivoyage.beta.wmflabs.org en.wiktionary.beta.wmflabs.org deployment.wikimedia.beta.wmflabs.org commons.wikimedia.beta.wmflabs.org api.wikimedia.beta.wmflabs.org login.wikimedia.beta.wmflabs.org; style-src 'self' data: blob: https://upload.wikimedia.beta.wmflabs.org upload.wikimedia.beta.wmflabs.org https://commons.wikimedia.beta.wmflabs.org https://upload.wikimedia.org https://commons.wikimedia.org meta.wikimedia.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org wikimedia.org 'unsafe-inline'; object-src 'none'; report-uri /w/api.php?action=cspreport&format=json vary: Accept-Encoding,Cookie,Authorization last-modified: Mon, 17 Oct 2022 11:50:33 GMT content-type: text/html; charset=UTF-8 age: 64 x-cache: deployment-cache-text07 miss, deployment-cache-text07 hit/4 x-cache-status: hit-remote server-timing: cache;desc="hit-remote", host;desc="deployment-cache-text07" report-to: { "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] } nel: { "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0} set-cookie: WMF-Last-Access=17-Oct-2022;Path=/;HttpOnly;secure;Expires=Fri, 18 Nov 2022 12:00:00 GMT set-cookie: WMF-Last-Access-Global=17-Oct-2022;Path=/;Domain=.wikipedia.beta.wmflabs.org;HttpOnly;secure;Expires=Fri, 18 Nov 2022 12:00:00 GMT accept-ch: Sec-CH-UA-Arch,Sec-CH-UA-Bitness,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-UA-Platform-Version permissions-policy: interest-cohort=(),ch-ua-arch=(self "intake-analytics.wikimedia.org"),ch-ua-bitness=(self "intake-analytics.wikimedia.org"),ch-ua-full-version-list=(self "intake-analytics.wikimedia.org"),ch-ua-model=(self "intake-analytics.wikimedia.org"),ch-ua-platform-version=(self "intake-analytics.wikimedia.org") x-client-ip: 127.0.0.1 cache-control: private, s-maxage=0, max-age=0, must-revalidate set-cookie: GeoIP=:::::v4; Path=/; secure; Domain=.beta.wmflabs.org accept-ranges: bytes content-length: 60787
vgutierrez@deployment-cache-text07:~$ sudo -i netstat --listen -n -p |grep 443 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 711/haproxy tcp6 0 0 :::443 :::* LISTEN 711/haproxy vgutierrez@deployment-cache-text07:~$ dpkg -l trafficserver Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-======================================================================== ii trafficserver 9.1.3-1wm2 amd64 fast, scalable and extensible HTTP/1.1 and HTTP/2.0 caching proxy server
deployment-cache-upload07 is up & running as well:
vgutierrez@deployment-cache-upload07:~$ curl -I --connect-to upload.wikimedia.beta.wmflabs.org:443:127.0.0.1 https://upload.wikimedia.beta.wmflabs.org/wikipedia/commons/d/de/123_4.jpg HTTP/2 200 content-length: 200456 x-object-meta-sha1base36: l3g1j3ermq3z1om07dpn18k7wq6dtis last-modified: Thu, 20 Jan 2022 19:42:40 GMT etag: 0d1cc1db7e0c7e206cd365a39bd14e12 content-type: image/jpeg date: Mon, 17 Oct 2022 13:37:06 GMT server: ATS/9.1.3 age: 4 x-cache: deployment-cache-upload07 miss, deployment-cache-upload07 hit/1 x-cache-status: hit-remote server-timing: cache;desc="hit-remote", host;desc="deployment-cache-upload07" report-to: { "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] } nel: { "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0} accept-ch: Sec-CH-UA-Arch,Sec-CH-UA-Bitness,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-UA-Platform-Version permissions-policy: interest-cohort=(),ch-ua-arch=(self "intake-analytics.wikimedia.org"),ch-ua-bitness=(self "intake-analytics.wikimedia.org"),ch-ua-full-version-list=(self "intake-analytics.wikimedia.org"),ch-ua-model=(self "intake-analytics.wikimedia.org"),ch-ua-platform-version=(self "intake-analytics.wikimedia.org") x-client-ip: 127.0.0.1 access-control-allow-origin: * access-control-expose-headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache timing-allow-origin: * accept-ranges: bytes
vgutierrez@deployment-cache-upload07:~$ dpkg -l trafficserver Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-======================================================================== ii trafficserver 9.1.3-1wm2 amd64 fast, scalable and extensible HTTP/1.1 and HTTP/2.0 caching proxy server vgutierrez@deployment-cache-upload07:~$ sudo -i netstat --listen -n -p |grep 443 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 734/haproxy tcp6 0 0 :::443 :::* LISTEN 734/haproxy
Mentioned in SAL (#wikimedia-releng) [2022-10-17T13:54:23Z] <vgutierrez> switch 185.15.56.35 from deployment-cache-upload06 to deployment-cache-upload07 - T320930
Mentioned in SAL (#wikimedia-releng) [2022-10-17T13:56:16Z] <vgutierrez> switch 185.15.56.36 from deployment-cache-text06 to deployment-cache-text07 - T320930
Change 843500 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] hieradata::deployment-prep: Bump deployment-cache-text|upload instances
Mentioned in SAL (#wikimedia-releng) [2022-10-17T14:01:58Z] <vgutierrez> shutdown deployment-cache-(text|upload)06 - T320930
Change 843500 merged by Vgutierrez:
[operations/puppet@production] hieradata::deployment-prep: Bump deployment-cache-text|upload instances
Mentioned in SAL (#wikimedia-releng) [2022-10-18T08:03:36Z] <vgutierrez> wipe deployment-cache-(text|upload)06 - T320930