Page MenuHomePhabricator
Create Task
Maniphest T309651

Package and deploy ATS 9.1.3
Closed, ResolvedPublic
Actions

Description

We are currently running ATS 8.0.8 in production and should upgrade to 9.1.3, which will be supported till 2024.

(As per https://docs.trafficserver.apache.org/en/latest/release-notes/roadmap.en.html, 8.x will be supported till 2022 and therefore we should not upgrade to the 8.x series.)

This task tracks the Debian packaging work and deployment of ATS 9.1.3.

  • Debian packaging
    • review packaging
  • 9.x config changes
    • review changes
  • Testing of new changes

Details

SubjectRepoBranchLines +/-
trafficserver: 9.x upgrade: separate metric current_client_connectionsoperations/puppetproduction+2 -0
Backport several fixes scheduled for 9.1.3operations/debs/trafficservermaster+395 -0
hiera: upgrade cp hosts in eqiad to ATS9operations/puppetproduction+1 -2
hiera: upgrade cp hosts in esams to ATS9operations/puppetproduction+1 -2
hiera: upgrade cp hosts in eqsin to ATS9operations/puppetproduction+1 -2
hiera: upgrade cp hosts in codfw to ATS9operations/puppetproduction+1 -0
hiera: upgrade cp hosts in ulsfo to ATS9operations/puppetproduction+0 -1
trafficserver: Update to ATS 9 on drmrsoperations/puppetproduction+1 -0
trafficserver: remove deprecated config for ATS9operations/puppetproduction+2 -0
trafficserver: Remove custom log for cp6008 and cp6016operations/puppetproduction+0 -16
hiera: downgrade cp6016 to ATS8operations/puppetproduction+0 -1
hiera: downgrade cp6008 to ATS8operations/puppetproduction+0 -1
trafficserver: Upgrade cp5014 and cp5016 to ATS 9operations/puppetproduction+2 -0
trafficserver: 9.x upgrade: install ATS 9.x from componentoperations/puppetproduction+18 -3
hiera: enable ATS9 on cp3064operations/puppetproduction+1 -0
hiera: enable ATS9 on cp1090operations/puppetproduction+1 -0
hiera: enable ATS9 on cp3065operations/puppetproduction+1 -0
hiera: enable ATS9 on cp1089operations/puppetproduction+1 -0
Release 9.1.3-1wm1operations/debs/trafficservermaster+5 -0
mtail: Fix trafficserver_backend_client_ttfb histogramoperations/puppetproduction+5 -53
mtail: Tune histogram buckets for trafficserver plugin time metricsoperations/puppetproduction+6 -4
trafficserver: Adjust (Total|Active)PluginTime milestonesoperations/puppetproduction+1 -1
trafficserver: Track time spent in or waiting for pluginsoperations/puppetproduction+32 -10
trafficserver: Set thread count factor to 1.5x for ats9 instancesoperations/puppetproduction+1 -0
hiera: enable ATS9 on cp6016operations/puppetproduction+1 -0
hiera: enable ATS9 on cp6008operations/puppetproduction+1 -0
trafficserver: 9.x upgrade: add compatibility for session_sharing.matchoperations/puppetproduction+92 -3
hiera: enable ATS9 on cp4032operations/puppetproduction+1 -0
C:trafficserver: filter out logs without ensure == presentoperations/puppetproduction+1 -1
C:trafficserver: build yaml structure in puppet instead of erboperations/puppetproduction+11 -43
trafficserver: Fix logging templateoperations/puppetproduction+37 -1
trafficserver: add top-level tag for loggingoperations/puppetproduction+3 -0
trafficserver: Avoid loading plugins from /runoperations/puppetproduction+3 -0
hiera: enable ATS9 on cp4026operations/puppetproduction+1 -0
trafficserver: 9.x upgrade: remove deprecated parent_proxy_routing_enableoperations/puppetproduction+2 -0
trafficserver: 9.x upgrade: replace client.verify.serveroperations/puppetproduction+24 -7
trafficserver: 9.x upgrade: switch ip_allow.config to YAML formatoperations/puppetproduction+41 -3
trafficserver: 9.x upgrade: rename max_connections_active_inoperations/puppetproduction+78 -62
trafficserver: 9.x upgrade: remove wmf-tls log formatoperations/puppetproduction+1 -3
trafficserver: 9.x upgrade: remove redundant metricsoperations/puppetproduction+0 -8
aptrepo: add a component for ATS 9.xoperations/puppetproduction+1 -0
trafficserver: 9.x upgrade: do not enable ATS 9.x by defaultoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jul 29 2022, 5:22 PM

Change 818504 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] trafficserver: 9.x upgrade: add compatibility for session_sharing.match

https://gerrit.wikimedia.org/r/818504

gerritbot added a comment.Aug 1 2022, 8:22 AM

Change 818504 merged by Vgutierrez:

[operations/puppet@production] trafficserver: 9.x upgrade: add compatibility for session_sharing.match

https://gerrit.wikimedia.org/r/818504

gerritbot added a comment.Aug 1 2022, 10:04 AM

Change 818456 merged by Vgutierrez:

[operations/puppet@production] hiera: enable ATS9 on cp6008

https://gerrit.wikimedia.org/r/818456

Stashbot added a comment.Aug 1 2022, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2022-08-01T10:05:28Z] <vgutierrez> test ATS 9.1.2 on cp6008 - T309651

gerritbot added a comment.Aug 1 2022, 10:09 AM

Change 818458 merged by Vgutierrez:

[operations/puppet@production] hiera: enable ATS9 on cp6016

https://gerrit.wikimedia.org/r/818458

Stashbot added a comment.Aug 1 2022, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2022-08-01T10:09:37Z] <vgutierrez> test ATS 9.1.2 on cp6016 - T309651

gerritbot added a comment.Aug 2 2022, 1:14 PM

Change 819585 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Set thread count factor to 1.5x for ats9 instances

https://gerrit.wikimedia.org/r/819585

gerritbot added a comment.Aug 2 2022, 1:23 PM

Change 819585 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Set thread count factor to 1.5x for ats9 instances

https://gerrit.wikimedia.org/r/819585

Stashbot added a comment.Aug 2 2022, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-02T13:24:44Z] <vgutierrez> restarting ATS 9.x instances to apply https://gerrit.wikimedia.org/r/819585 - T309651

gerritbot added a comment.Aug 3 2022, 11:21 AM

Change 820104 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Track time spent in or waiting for plugins

https://gerrit.wikimedia.org/r/820104

gerritbot added a comment.Aug 3 2022, 1:00 PM

Change 820121 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/debs/trafficserver@master] Backport several fixex scheduled for 9.1.3

https://gerrit.wikimedia.org/r/820121

gerritbot added a comment.Aug 3 2022, 1:32 PM

Change 820104 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Track time spent in or waiting for plugins

https://gerrit.wikimedia.org/r/820104

gerritbot added a comment.Aug 3 2022, 1:43 PM

Change 820132 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Adjust (Total|Active)PluginTime milestones

https://gerrit.wikimedia.org/r/820132

gerritbot added a comment.Aug 3 2022, 1:47 PM

Change 820132 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Adjust (Total|Active)PluginTime milestones

https://gerrit.wikimedia.org/r/820132

Stashbot added a comment.Aug 3 2022, 3:30 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-03T15:30:54Z] <vgutierrez> clearing ats-be cache on cp6016 - T309651

Stashbot added a comment.Aug 3 2022, 3:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-03T15:38:50Z] <vgutierrez> clearing ats-be cache on cp6008 - T309651

Stashbot added a comment.Aug 9 2022, 5:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-09T17:29:56Z] <vgutierrez> test trafficserver 9.1.2-1wm2 in cp6016 - T309651

gerritbot added a comment.Aug 10 2022, 10:21 AM

Change 822054 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] mtail: Tune histogram buckets for trafficserver plugin time metrics

https://gerrit.wikimedia.org/r/822054

gerritbot added a comment.Aug 10 2022, 10:53 AM

Change 822054 merged by Vgutierrez:

[operations/puppet@production] mtail: Tune histogram buckets for trafficserver plugin time metrics

https://gerrit.wikimedia.org/r/822054

gerritbot added a comment.Aug 10 2022, 1:13 PM

Change 822078 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/debs/trafficserver@master] Release 9.1.3-1wm1

https://gerrit.wikimedia.org/r/822078

gerritbot added a comment.Aug 10 2022, 1:27 PM

Change 822080 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] mtail: Fix trafficserver_backend_client_ttfb histogram

https://gerrit.wikimedia.org/r/822080

gerritbot added a comment.Aug 10 2022, 2:24 PM

Change 822080 merged by Vgutierrez:

[operations/puppet@production] mtail: Fix trafficserver_backend_client_ttfb histogram

https://gerrit.wikimedia.org/r/822080

gerritbot added a comment.Aug 10 2022, 2:58 PM

Change 822078 merged by Vgutierrez:

[operations/debs/trafficserver@master] Release 9.1.3-1wm1

https://gerrit.wikimedia.org/r/822078

Stashbot added a comment.Aug 10 2022, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T16:13:56Z] <sukhe> reprepro -C component/trafficserver9 include buster-wikimedia trafficserver_9.1.3-1wm1_amd64.changes: T309651

Stashbot added a comment.Aug 10 2022, 4:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T16:56:30Z] <sukhe> testing ATS 9.1.3-1wm1 on cp6016: T309651

Stashbot added a comment.Aug 10 2022, 5:02 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T17:02:07Z] <sukhe> testing ATS 9.1.3-1wm1 on cp6008: T309651

Stashbot added a comment.Aug 10 2022, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T17:06:53Z] <sukhe> testing ATS 9.1.3-1wm1 on cp4032: T309651

Stashbot added a comment.Aug 10 2022, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T19:28:28Z] <sukhe> testing ATS 9.1.3-1wm1 on cp4026: T309651

gerritbot added a comment.Aug 11 2022, 2:17 PM

Change 822381 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: enable ATS9 on cp1089

https://gerrit.wikimedia.org/r/822381

gerritbot added a comment.Aug 11 2022, 2:20 PM

Change 822382 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: enable ATS9 on cp1090

https://gerrit.wikimedia.org/r/822382

gerritbot added a comment.Aug 11 2022, 2:22 PM

Change 822384 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: enable ATS9 on cp3064

https://gerrit.wikimedia.org/r/822384

gerritbot added a comment.Aug 11 2022, 2:23 PM

Change 822406 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: enable ATS9 on cp3065

https://gerrit.wikimedia.org/r/822406

gerritbot added a comment.Aug 11 2022, 5:22 PM

Change 822381 merged by Ssingh:

[operations/puppet@production] hiera: enable ATS9 on cp1089

https://gerrit.wikimedia.org/r/822381

Stashbot added a comment.Aug 11 2022, 5:28 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-11T17:28:00Z] <sukhe> testing ATS 9.1.3-1wm1 on cp1089: T309651

gerritbot added a comment.Aug 11 2022, 5:31 PM

Change 822406 merged by Ssingh:

[operations/puppet@production] hiera: enable ATS9 on cp3065

https://gerrit.wikimedia.org/r/822406

Stashbot added a comment.Aug 11 2022, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-11T17:33:08Z] <sukhe> testing ATS 9.1.3-1wm1 on cp3065: T309651

gerritbot added a comment.Aug 11 2022, 5:36 PM

Change 822382 merged by Ssingh:

[operations/puppet@production] hiera: enable ATS9 on cp1090

https://gerrit.wikimedia.org/r/822382

Stashbot added a comment.Aug 11 2022, 5:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-11T17:38:44Z] <sukhe> testing ATS 9.1.3-1wm1 on cp1090: T309651

gerritbot added a comment.Aug 11 2022, 5:41 PM

Change 822384 merged by Ssingh:

[operations/puppet@production] hiera: enable ATS9 on cp3064

https://gerrit.wikimedia.org/r/822384

Stashbot added a comment.Aug 11 2022, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-11T17:52:26Z] <sukhe> testing ATS 9.1.3-1wm1 on cp3064: T309651

ssingh renamed this task from Package and deploy ATS 9.1.2 to Package and deploy ATS 9.1.3.Aug 12 2022, 2:40 PM
ssingh updated the task description. (Show Details)
TheresNoTime mentioned this in T315351: Evaluation Error on deployment-cache-text06 puppet run .Aug 16 2022, 11:30 PM
TheresNoTime subscribed.
gerritbot added a comment.Aug 18 2022, 8:30 AM

Change 824408 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Upgrade cp5014 and cp5016 to ATS 9

https://gerrit.wikimedia.org/r/824408

gerritbot added a comment.Aug 18 2022, 8:32 AM

Change 824408 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Upgrade cp5014 and cp5016 to ATS 9

https://gerrit.wikimedia.org/r/824408

Stashbot added a comment.Aug 18 2022, 8:33 AM

Mentioned in SAL (#wikimedia-operations) [2022-08-18T08:33:28Z] <vgutierrez> upgrade to ATS 9.1.3 in cp5014 and cp5016 - T309651

MoritzMuehlenhoff subscribed.Aug 22 2022, 4:14 PM
BCornwall subscribed.Aug 24 2022, 6:45 PM
BCornwall mentioned this in T292815: ATS should alert if the number of total or active connections reached maximum.Aug 24 2022, 7:20 PM
ssingh added a comment.Aug 31 2022, 2:06 PM

Downgrading and reimaging drmrs ATS9 hosts cp6008 and cp6016 to ATS8 for a week so that we can have comparative data later when we upgrade all instances to ATS9 in drmrs.

Krinkle awarded a token.Aug 31 2022, 2:06 PM
Krinkle added a project: Performance-Team (Radar).
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
ops-monitoring-bot added a comment.Aug 31 2022, 2:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6008.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Aug 31 2022, 2:09 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6008.drmrs.wmnet with OS buster executed with errors:

  • cp6008 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp6008.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_upload,service=ats-tls"}
{"cp6008.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_upload,service=varnish-fe"}
{"cp6008.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_upload,service=ats-be"}

  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_upload,service=ats-tls' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_upload,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_upload,service=ats-be' set/pooled=yes

  • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Aug 31 2022, 2:13 PM

Change 828540 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: downgrade cp6008 to ATS8

https://gerrit.wikimedia.org/r/828540

gerritbot added a comment.Aug 31 2022, 2:15 PM

Change 828540 merged by Ssingh:

[operations/puppet@production] hiera: downgrade cp6008 to ATS8

https://gerrit.wikimedia.org/r/828540

ops-monitoring-bot added a comment.Aug 31 2022, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6008.drmrs.wmnet with OS buster

gerritbot added a comment.Aug 31 2022, 2:18 PM

Change 828543 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: downgrade cp6016 to ATS8

https://gerrit.wikimedia.org/r/828543

ops-monitoring-bot added a comment.Aug 31 2022, 3:00 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6008.drmrs.wmnet with OS buster completed:

  • cp6008 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp6008.drmrs.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=drmrs,cluster=cache_upload,service=ats-be"}
{"cp6008.drmrs.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=drmrs,cluster=cache_upload,service=ats-tls"}
{"cp6008.drmrs.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=drmrs,cluster=cache_upload,service=varnish-fe"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208311415_sukhe_3601317_cp6008.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_upload,service=ats-be' set/pooled=no
sudo confctl select 'dc=drmrs,cluster=cache_upload,service=ats-tls' set/pooled=no
sudo confctl select 'dc=drmrs,cluster=cache_upload,service=varnish-fe' set/pooled=no

  • Updated Netbox data from PuppetDB
gerritbot added a comment.Aug 31 2022, 3:06 PM

Change 828543 merged by Ssingh:

[operations/puppet@production] hiera: downgrade cp6016 to ATS8

https://gerrit.wikimedia.org/r/828543

ops-monitoring-bot added a comment.Aug 31 2022, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6016.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Aug 31 2022, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6016.drmrs.wmnet with OS buster completed:

  • cp6016 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp6016.drmrs.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6016.drmrs.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6016.drmrs.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208311507_sukhe_3611351_cp6016.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=no
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=no
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=no

  • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 1 2022, 8:08 AM

Change 828758 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Remove custom log for cp6008 and cp6016

https://gerrit.wikimedia.org/r/828758

gerritbot added a comment.Sep 1 2022, 8:11 AM

Change 828758 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Remove custom log for cp6008 and cp6016

https://gerrit.wikimedia.org/r/828758

gerritbot added a comment.Sep 1 2022, 2:46 PM

Change 829022 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] trafficserver: remove deprecated config for ATS9

https://gerrit.wikimedia.org/r/829022

gerritbot added a comment.Sep 1 2022, 3:06 PM

Change 829022 merged by Ssingh:

[operations/puppet@production] trafficserver: remove deprecated config for ATS9

https://gerrit.wikimedia.org/r/829022

Vgutierrez triaged this task as Medium priority.Sep 7 2022, 9:33 AM
gerritbot added a comment.Sep 8 2022, 1:36 PM

Change 830867 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Update to ATS 9 on drmrs

https://gerrit.wikimedia.org/r/830867

Stashbot added a comment.Sep 8 2022, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-08T13:39:46Z] <vgutierrez> disable puppet on A:cp-drmrs during the update to ATS 9.1.3 - T309651

gerritbot added a comment.Sep 8 2022, 1:41 PM

Change 830867 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Update to ATS 9 on drmrs

https://gerrit.wikimedia.org/r/830867

Stashbot added a comment.Sep 8 2022, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-08T13:43:13Z] <vgutierrez> rolling upgrade to ats 9 in cp drmrs - T309651

Vgutierrez closed subtask T315064: metric discrepancies between ATS 9.x and ATS 8.x as Resolved.Sep 9 2022, 10:38 AM
gerritbot added a comment.Oct 3 2022, 1:31 PM

Change 837670 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: upgrade cp hosts in codfw to ATS9

https://gerrit.wikimedia.org/r/837670

gerritbot added a comment.Oct 3 2022, 1:38 PM

Change 837673 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: upgrade cp hosts in ulsfo to ATS9

https://gerrit.wikimedia.org/r/837673

Stashbot added a comment.Oct 3 2022, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-03T13:57:49Z] <sukhe> reprepro -C component/trafficserver9 include buster-wikimedia trafficserver_9.1.3-1wm2_amd64.changes: T309651

Stashbot added a comment.Oct 3 2022, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-03T14:08:30Z] <sukhe> upgrade cp4026, cp4032 to ATS 9.1.3-1wm2 from 9.1.3-1wm1: T309651

Stashbot added a comment.Oct 3 2022, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-03T14:35:21Z] <sukhe> upgrade A:cp and A:drmrs to ATS 9.1.3-1wm2 from 9.1.3-1wm1: T309651

Stashbot added a comment.Oct 3 2022, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-03T15:15:55Z] <sukhe> disable Puppet on cp hosts in ulsfo: rolling out T309651

gerritbot added a comment.Oct 3 2022, 3:17 PM

Change 837673 merged by Ssingh:

[operations/puppet@production] hiera: upgrade cp hosts in ulsfo to ATS9

https://gerrit.wikimedia.org/r/837673

Stashbot added a comment.Oct 3 2022, 4:14 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-03T16:14:54Z] <sukhe> disable Puppet on cp hosts in codfw: rolling out T309651

gerritbot added a comment.Oct 3 2022, 4:16 PM

Change 837670 merged by Ssingh:

[operations/puppet@production] hiera: upgrade cp hosts in codfw to ATS9

https://gerrit.wikimedia.org/r/837670

ssingh added a comment.Oct 3 2022, 5:04 PM

We are running ATS9 on all cp hosts in: codfw, ulsfo, drmrs, in addition to the existing hosts in eqiad, esams, eqsin, the site-wide deployment of which will continue tomorrow (51/92 hosts as of today).

Hosts running ATS9 (9.1.3-1wm1 and 9.1.3-1wm2):

cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1089-1090].eqiad.wmnet,cp[5014,5016].eqsin.wmnet,cp[3064-3065].esams.wmnet,cp[4022-4026,4028-4030,4032-4036].ulsfo.wmnet

The hosts running ATS 9.1.3-1wm1 and that will be upgraded to 9.1.3-1wm2 as part of their site upgrade:

cp1089.eqiad.wmnet, cp1090.eqiad.wmnet, cp3064.esams.wmnet, cp3065.esams.wmnet, cp5014.eqsin.wmnet, cp5016.eqsin.wmnet
gerritbot added a comment.Oct 4 2022, 1:34 PM

Change 838149 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: upgrade cp hosts in eqsin to ATS9

https://gerrit.wikimedia.org/r/838149

gerritbot added a comment.Oct 4 2022, 1:42 PM

Change 838154 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: upgrade cp hosts in esams to ATS9

https://gerrit.wikimedia.org/r/838154

gerritbot added a comment.Oct 4 2022, 1:44 PM

Change 838155 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: upgrade cp hosts in eqiad to ATS9

https://gerrit.wikimedia.org/r/838155

Stashbot added a comment.Oct 4 2022, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-04T13:48:44Z] <sukhe> disable Puppet on A:cp and A:eqsin for T309651

gerritbot added a comment.Oct 4 2022, 1:49 PM

Change 838149 merged by Ssingh:

[operations/puppet@production] hiera: upgrade cp hosts in eqsin to ATS9

https://gerrit.wikimedia.org/r/838149

Stashbot added a comment.Oct 4 2022, 2:51 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-04T14:51:35Z] <sukhe> disable Puppet on A:cp and A:esams for T309651

gerritbot added a comment.Oct 4 2022, 2:54 PM

Change 838154 merged by Ssingh:

[operations/puppet@production] hiera: upgrade cp hosts in esams to ATS9

https://gerrit.wikimedia.org/r/838154

Stashbot added a comment.Oct 4 2022, 3:47 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-04T15:47:55Z] <sukhe> disable Puppet on A:cp and A:eqiad for T309651

gerritbot added a comment.Oct 4 2022, 5:06 PM

Change 838155 merged by Ssingh:

[operations/puppet@production] hiera: upgrade cp hosts in eqiad to ATS9

https://gerrit.wikimedia.org/r/838155

ssingh closed this task as Resolved.Oct 4 2022, 5:51 PM
sukhe@cumin2002:~$ sudo cumin 'A:cp' '/usr/bin/traffic_server --version'
92 hosts will be targeted:
cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1075-1090].eqiad.wmnet,cp[5002-5016].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4022-4026,4028-4030,4032-4036].ulsfo.wmnet
Ok to proceed on 92 hosts? Enter the number of affected hosts to confirm or "q" to quit 92
===== NODE GROUP =====                                                                                                                
(92) cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1075-1090].eqiad.wmnet,cp[5002-5016].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4022-4026,4028-4030,4032-4036].ulsfo.wmnet                                                                                        
----- OUTPUT of '/usr/bin/traffic_server --version' -----                                                                             
Traffic Server 9.1.3 Sep 20 2022 15:57:00 localhost                                                                                   
traffic_server: using root directory '/usr'                                                                                           
Apache Traffic Server - traffic_server - 9.1.3 - (build # 092015 on Sep 20 2022 at 15:57:00)
================                                                                                                                      
PASS |██████████████████████████████████████████████████████████████████████████████████████| 100% (92/92) [00:03<00:00, 23.70hosts/s]

We have upgraded all cp hosts to ATS9 and resolving this task. Thanks to everyone involved.

ssingh updated the task description. (Show Details)Oct 4 2022, 5:51 PM
Vgutierrez mentioned this in T321776: Clean up after ATS 9.x upgrade.Oct 27 2022, 9:14 AM
gerritbot added a comment.Oct 28 2022, 8:25 AM

Change 820121 abandoned by Vgutierrez:

[operations/debs/trafficserver@master] Backport several fixes scheduled for 9.1.3

Reason:

we already moved to 9.1.3

https://gerrit.wikimedia.org/r/820121

Vgutierrez closed subtask T321776: Clean up after ATS 9.x upgrade as Resolved.Nov 2 2022, 2:14 PM
gerritbot added a comment.Nov 9 2022, 9:23 AM

Change 803285 merged by Vgutierrez:

[operations/puppet@production] trafficserver: 9.x upgrade: separate metric current_client_connections

https://gerrit.wikimedia.org/r/803285

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL