After the successful production switchover to ATS performed in T213263, we can now begin replacing Varnish backends with ATS on cache nodes, beginning with the upload cluster in ulsfo.
As a first step, we are going to produce the necessary puppetization to deploy a mixed varnish-fe/ats-be cache node. Then we are going to use such puppetization to reimage existing nodes. Note that ATS backends will be routed straight to the applayer, rather than through other DCs.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • ema | T226589 Replace Varnish backends with ATS on cache upload nodes | |||
| Resolved | • ema | T219967 Replace Varnish backends with ATS on cache upload nodes in ulsfo |
Change 504895 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: remove unused hiera settings
Change 504895 merged by Ema:
[operations/puppet@production] cache: remove unused purge-related hiera settings
Change 505708 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] conftool-data: add ats-be to cache_upload@ulsfo
Change 505708 merged by Ema:
[operations/puppet@production] conftool-data: add ats-be to cache_upload@ulsfo
Change 505724 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: stop passing route_table to varnish-fe
Change 505724 merged by Ema:
[operations/puppet@production] cache: stop passing route_table to varnish-fe
Change 505748 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] profile::trafficserver::backend: do not configure vhtcpd
Change 505748 merged by Ema:
[operations/puppet@production] profile::trafficserver::backend: do not configure vhtcpd
Change 505767 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: reimage cp4021 as upload_ats
Change 501360 merged by Ema:
[operations/puppet@production] role::cache::upload_ats: Varnish frontend / ATS backend setup
Mentioned in SAL (#wikimedia-operations) [2019-04-23T13:54:48Z] <ema> depool cp4021 and reimage as upload_ats T219967
Change 505767 merged by Ema:
[operations/puppet@production] cache: reimage cp4021 as upload_ats
Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:
['cp4021.ulsfo.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201904231359_ema_129175.log.
Change 505789 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: add ATS hiera settings to role upload_ats
Change 505789 merged by Ema:
[operations/puppet@production] cache: add ATS hiera settings to role upload_ats
Change 505799 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] trafficserver: run apt-get update before installing
Change 505799 merged by Ema:
[operations/puppet@production] trafficserver: run apt-get update before installing
Change 505805 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] trafficserver: avoid apt dependency cycle
Change 505805 merged by Ema:
[operations/puppet@production] trafficserver: avoid apt dependency cycle
Change 505815 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: distinguish between upload and upload_ats nodes
Change 506090 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move check_varnish_expiry_mailbox_lag to backend profile
Change 506090 merged by Ema:
[operations/puppet@production] cache: move check_varnish_expiry_mailbox_lag to backend profile
Change 506122 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: use ATS profile instead of role in job definition
Change 506122 merged by Ema:
[operations/puppet@production] prometheus: use ATS profile instead of role in job definition
Change 506154 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: unify cache nodes definition in hieradata
Change 506154 merged by Ema:
[operations/puppet@production] cache: unify cache nodes definition in hieradata
Change 506177 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cumin: add ATS production hosts to aliases
Change 506177 merged by Ema:
[operations/puppet@production] cumin: add ATS production hosts to aliases
Change 505815 merged by Ema:
[operations/puppet@production] cache: distinguish between Varnish and ATS nodes
Change 506389 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move varnish etcd-based directors to profile
Change 506409 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: add reload_vcl_opts function
Change 506409 merged by Ema:
[operations/puppet@production] varnish: add reload_vcl_opts function
Change 506389 merged by Ema:
[operations/puppet@production] cache: move varnish etcd-based directors to profile
Change 506445 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] conftool-data: cp4021 only ats-be in production
Change 506480 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: multiple keyspaces support for directors.frontend.vcl
Change 506484 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: do not set backend_service
Change 506445 merged by Ema:
[operations/puppet@production] conftool-data: set cp4021 as the only ats-be in production
Change 506480 merged by Ema:
[operations/puppet@production] cache: multiple keyspaces support for directors.frontend.vcl
Change 506624 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] conftool-data: define ats-be for text/upload in all DCs
Change 506624 merged by Ema:
[operations/puppet@production] conftool-data: define ats-be for text/upload in all DCs
Mentioned in SAL (#wikimedia-operations) [2019-04-26T10:19:09Z] <ema> depool cp3030 for testing T219967
Change 506633 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Revert "conftool-data: define ats-be for text/upload in all DCs"
Change 506633 merged by Ema:
[operations/puppet@production] Revert "conftool-data: define ats-be for text/upload in all DCs"
Change 506636 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: hiera setting to list backend services
Change 506484 abandoned by Ema:
cache: do not set backend_service
Reason:
Obsoleted by https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /506636/
Change 506636 merged by Ema:
[operations/puppet@production] cache: hiera setting to list backend services
Mentioned in SAL (#wikimedia-operations) [2019-04-26T12:20:36Z] <ema> repool cp3030 after directors.frontend.vcl testing T219967
Mentioned in SAL (#wikimedia-operations) [2019-04-26T12:44:43Z] <ema> pool cp4021 w/ ATS backend T219967
Change 506980 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cumin aliases: upload_ats is upload
Change 506980 merged by Ema:
[operations/puppet@production] cumin aliases: upload_ats is upload
Change 507022 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Add profile::cache::varnish::frontend::text
Change 507022 merged by Ema:
[operations/puppet@production] Add profile::cache::varnish::frontend::text
Change 507266 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: reimage cp4022 as upload_ats
Change 507267 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: add ulsfo_ats to cacheproxy::cron_restart
Change 507267 merged by Ema:
[operations/puppet@production] cache: add ATS nodes to cacheproxy::cron_restart
Mentioned in SAL (#wikimedia-operations) [2019-04-30T13:28:19Z] <ema> depool cp4022 and reimage as upload_ats T219967
Change 507266 merged by Ema:
[operations/puppet@production] cache: reimage cp4022 as upload_ats
Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:
['cp4022.ulsfo.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201904301334_ema_110727.log.
Completed auto-reimage of hosts:
['cp4022.ulsfo.wmnet']
Of which those FAILED:
['cp4022.ulsfo.wmnet']
Mentioned in SAL (#wikimedia-operations) [2019-04-30T16:04:50Z] <ema> pool cp4022 w/ ATS backend T219967
Change 507358 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: add hiera setting for varnish backend restarts
Change 507358 merged by Ema:
[operations/puppet@production] cache: add hiera setting for varnish backend restarts
Change 507744 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: reimage cp4023 as upload_ats
Mentioned in SAL (#wikimedia-operations) [2019-05-02T09:02:48Z] <ema> depool cp4023 and reimage as upload_ats T219967
Change 507744 merged by Ema:
[operations/puppet@production] cache: reimage cp4023 as upload_ats
Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:
['cp4023.ulsfo.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201905020907_ema_2088.log.
Mentioned in SAL (#wikimedia-operations) [2019-05-02T10:03:58Z] <ema> pool cp4023 w/ ATS backend T219967
Change 507915 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: reimage cp4024 as upload_ats
Mentioned in SAL (#wikimedia-operations) [2019-05-03T07:45:52Z] <ema> depool cp4024 and reimage as upload_ats T219967
Change 507915 merged by Ema:
[operations/puppet@production] cache: reimage cp4024 as upload_ats
Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:
['cp4024.ulsfo.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201905030750_ema_2659.log.
Mentioned in SAL (#wikimedia-operations) [2019-05-03T08:47:40Z] <ema> pool cp4024 w/ ATS backend T219967
Change 507935 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: reimage cp4025 as upload_ats
Mentioned in SAL (#wikimedia-operations) [2019-05-03T09:49:27Z] <ema> depool cp4025 and reimage as upload_ats T219967
Change 507935 merged by Ema:
[operations/puppet@production] cache: reimage cp4025 as upload_ats
Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:
['cp4025.ulsfo.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201905030954_ema_30037.log.
Mentioned in SAL (#wikimedia-operations) [2019-05-03T10:47:49Z] <ema> pool cp4025 w/ ATS backend T219967
Change 507953 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: retry requests upon 502 errors
Change 508284 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: add upload_ats target
Change 508284 merged by Ema:
[operations/puppet@production] prometheus: add upload_ats mtail targets
Change 508296 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: add glob for ATS to file_sd_configs
Change 508296 merged by Ema:
[operations/puppet@production] prometheus: add glob for ATS to file_sd_configs
Change 508304 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: reimage cp4026 as upload_ats
Mentioned in SAL (#wikimedia-operations) [2019-05-06T14:19:52Z] <ema> depool cp4026 and reimage as upload_ats T219967
Change 508304 merged by Ema:
[operations/puppet@production] cache: reimage cp4026 as upload_ats
Script wmf-auto-reimage was launched by ema on cumin2001.codfw.wmnet for hosts:
['cp4026.ulsfo.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201905061425_ema_2856.log.
Mentioned in SAL (#wikimedia-operations) [2019-05-06T15:14:31Z] <ema> pool cp4026 w/ ATS backend T219967
Change 507953 abandoned by Ema:
varnish: retry requests upon 502 errors
Reason:
This isn't necessary anymore:
https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /525222/