Page MenuHomePhabricator
Create Task
Maniphest T247963

Migrate role::graphite::production to Bullseye
Closed, ResolvedPublic
Actions

Description

As per title, all hosts running Graphite should be on Bullseye

root@cumin1001:~# cumin 'P{C:graphite} and not P{F:lsbdistcodename = buster}'
2 hosts will be targeted:
graphite2003.codfw.wmnet,graphite1004.eqiad.wmnet
DRY-RUN mode enabled, aborting
  • Get the graphite::production role to work in Pontoon on Bullseye

Action plan for codfw:

The plan for eqiad is similar, with the addition of a failover to codfw as per https://wikitech.wikimedia.org/wiki/Graphite#Failover and fail back once things are working in eqiad.

Details

SubjectRepoBranchLines +/-
graphite: bump fetch_timeoutoperations/puppetproduction+10 -0
Revert "ProductionServices: use graphite2003 for statsd"operations/mediawiki-configmaster+2 -2
Revert "wmnet: move writes to graphite2003"operations/dnsmaster+3 -3
Revert "statsd: failover writes to graphite2003"operations/puppetproduction+1 -1
Revert "monitoring: check graphite2003 metrics"operations/puppetproduction+3 -3
Revert "discovery: move read traffic to graphite2003"operations/dnsmaster+2 -2
graphite: set CLUSTER_SERVERS empty with no remote serversoperations/puppetproduction+4 -6
mwdebug: fix statsd network policyoperations/deployment-chartsmaster+6 -1
install_server: use standard recipe for all graphite hostsoperations/puppetproduction+1 -67
ProductionServices: use graphite2003 for statsdoperations/mediawiki-configmaster+2 -2
wmnet: move writes to graphite2003operations/dnsmaster+3 -3
statsd: failover writes to graphite2003operations/puppetproduction+1 -1
monitoring: check graphite2003 metricsoperations/puppetproduction+3 -3
discovery: move read traffic to graphite2003operations/dnsmaster+2 -2
graphite: expire metric files not updated for 3yoperations/puppetproduction+6 -0
graphite: disable tags supportoperations/puppetproduction+1 -0
graphite: move production to /srv/carbon as storage directoryoperations/puppetproduction+16 -4
install_server: use standard recipe for graphite2003operations/puppetproduction+1 -0
statsite: log instance identifieroperations/puppetproduction+1 -0
graphite: set settings_module from uwsgioperations/puppetproduction+2 -0
statsite: switch to python3 on Bullseyeoperations/puppetproduction+7 -1
pontoon: use graphite-04 in o11y stackoperations/puppetproduction+1 -1
graphite: add Bullseye version of graphite auth/indexoperations/puppetproduction+103 -0
graphite: add Bullseye supportoperations/puppetproduction+21 -11
graphite: stop using LVM for /srv in labsoperations/puppetproduction+0 -5
Fix installation of graphite-web on Busteroperations/puppetproduction+7 -4
graphite: django 2.2 compatoperations/puppetproduction+5 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
lmata edited projects, added SRE Observability (FY2021/2022-Q1); removed Observability-Metrics.Aug 24 2021, 3:28 PM
lmata renamed this task from Migrate role::graphite::production to Buster to Migrate role::graphite::production to Bullseye.Aug 31 2021, 4:19 PM
lmata edited projects, added SRE Observability (FY2021/2022-Q2); removed SRE Observability (FY2021/2022-Q1).
fgiunchedi updated the task description. (Show Details)Aug 31 2021, 4:21 PM
lmata triaged this task as Medium priority.Sep 30 2021, 9:48 PM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2021/2022-Q2) board.
gerritbot added a comment.Oct 5 2021, 12:44 PM

Change 726612 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: add Bullseye support

https://gerrit.wikimedia.org/r/726612

gerritbot added a project: Patch-For-Review.Oct 5 2021, 12:44 PM

Change 726613 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: add Bullseye version of graphite auth/index

https://gerrit.wikimedia.org/r/726613

gerritbot added a comment.Oct 5 2021, 12:44 PM

Change 726614 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: stop using LVM for /srv in labs

https://gerrit.wikimedia.org/r/726614

gerritbot added a comment.Oct 5 2021, 2:22 PM

Change 726614 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: stop using LVM for /srv in labs

https://gerrit.wikimedia.org/r/726614

gerritbot added a comment.Oct 5 2021, 2:31 PM

Change 726612 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: add Bullseye support

https://gerrit.wikimedia.org/r/726612

gerritbot added a comment.Oct 6 2021, 6:21 AM

Change 726613 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: add Bullseye version of graphite auth/index

https://gerrit.wikimedia.org/r/726613

gerritbot added a comment.Oct 6 2021, 6:41 AM

Change 726750 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] pontoon: use graphite-04 in o11y stack

https://gerrit.wikimedia.org/r/726750

gerritbot added a comment.Oct 6 2021, 6:43 AM

Change 726750 merged by Filippo Giunchedi:

[operations/puppet@production] pontoon: use graphite-04 in o11y stack

https://gerrit.wikimedia.org/r/726750

fgiunchedi added a project: User-fgiunchedi.Oct 6 2021, 7:16 AM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
fgiunchedi updated the task description. (Show Details)Oct 6 2021, 8:26 AM
fgiunchedi added a comment.Oct 6 2021, 8:28 AM

A few roadblocks and bugs but overall progress, so far:

  • graphite-web isn't in stable, I've imported the testing version to bullseye-wikimedia
  • carbon-c-relay has a CPU-hogging bug in stable, I've imported the testing version to bullseye-wikimedia
  • statsite needed an update (upstream version and python3). It is a local package and a new version lives in bullseye-wikimedia
fgiunchedi moved this task from Up next to In progress on the SRE Observability (FY2021/2022-Q2) board.Oct 6 2021, 2:24 PM
gerritbot added a comment.Oct 7 2021, 11:41 AM

Change 727293 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] statsite: switch to python3 on Bullseye

https://gerrit.wikimedia.org/r/727293

gerritbot added a comment.Oct 7 2021, 11:41 AM

Change 727294 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: set settings_module from uwsgi

https://gerrit.wikimedia.org/r/727294

gerritbot added a comment.Oct 7 2021, 11:41 AM

Change 727295 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] statsite: log instance identifier

https://gerrit.wikimedia.org/r/727295

gerritbot added a comment.Oct 7 2021, 1:52 PM

Change 727293 merged by Filippo Giunchedi:

[operations/puppet@production] statsite: switch to python3 on Bullseye

https://gerrit.wikimedia.org/r/727293

gerritbot added a comment.Oct 7 2021, 1:54 PM

Change 727294 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: set settings_module from uwsgi

https://gerrit.wikimedia.org/r/727294

fgiunchedi updated the task description. (Show Details)Oct 11 2021, 8:28 AM
gerritbot added a comment.Oct 11 2021, 8:49 AM

Change 727295 merged by Filippo Giunchedi:

[operations/puppet@production] statsite: log instance identifier

https://gerrit.wikimedia.org/r/727295

ops-monitoring-bot added a comment.Oct 11 2021, 9:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host graphite2003.codfw.wmnet

gerritbot added a comment.Oct 11 2021, 9:29 AM

Change 729934 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: use standard recipe for graphite2003

https://gerrit.wikimedia.org/r/729934

gerritbot added a comment.Oct 11 2021, 9:30 AM

Change 729934 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: use standard recipe for graphite2003

https://gerrit.wikimedia.org/r/729934

ops-monitoring-bot added a comment.Oct 11 2021, 9:45 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host graphite2003.codfw.wmnet completed:

  • graphite2003 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110110913_filippo_4002_graphite2003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Oct 11 2021, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host graphite2003.codfw.wmnet

ops-monitoring-bot added a comment.Oct 11 2021, 10:23 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host graphite2003.codfw.wmnet completed:

  • graphite2003 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110110950_filippo_29925_graphite2003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
fgiunchedi updated the task description. (Show Details)Oct 11 2021, 10:28 AM
gerritbot added a comment.Oct 11 2021, 11:57 AM

Change 729968 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: disable tags support

https://gerrit.wikimedia.org/r/729968

gerritbot added a comment.Oct 11 2021, 12:21 PM

Change 729975 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: move production to /srv/carbon as storage directory

https://gerrit.wikimedia.org/r/729975

gerritbot added a comment.Oct 13 2021, 8:10 AM

Change 729975 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: move production to /srv/carbon as storage directory

https://gerrit.wikimedia.org/r/729975

gerritbot added a comment.Oct 13 2021, 8:41 AM

Change 730427 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: expire metric files not updated for 3y

https://gerrit.wikimedia.org/r/730427

gerritbot added a comment.Oct 14 2021, 7:51 AM

Change 729968 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: disable tags support

https://gerrit.wikimedia.org/r/729968

gerritbot added a comment.Oct 18 2021, 8:32 AM

Change 730427 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: expire metric files not updated for 3y

https://gerrit.wikimedia.org/r/730427

Stashbot added a comment.Oct 18 2021, 9:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-18T09:38:04Z] <godog> sync metrics from graphite1004 to graphite2003 - T247963

gerritbot added a comment.Oct 18 2021, 1:02 PM

Change 731433 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] statsd: failover writes to graphite2003

https://gerrit.wikimedia.org/r/731433

gerritbot added a comment.Oct 18 2021, 1:02 PM

Change 731434 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] monitoring: check graphite2003 metrics

https://gerrit.wikimedia.org/r/731434

gerritbot added a comment.Oct 18 2021, 1:03 PM

Change 731435 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] discovery: move read traffic to graphite2003

https://gerrit.wikimedia.org/r/731435

gerritbot added a comment.Oct 18 2021, 1:03 PM

Change 731436 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] wmnet: move writes to graphite2003

https://gerrit.wikimedia.org/r/731436

gerritbot added a comment.Oct 19 2021, 8:48 AM

Change 731435 merged by Filippo Giunchedi:

[operations/dns@master] discovery: move read traffic to graphite2003

https://gerrit.wikimedia.org/r/731435

Stashbot added a comment.Oct 19 2021, 8:50 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T08:50:22Z] <godog> point graphite.discovery.wmnet to graphite2003 - T247963

gerritbot added a comment.Oct 19 2021, 9:18 AM

Change 731434 merged by Filippo Giunchedi:

[operations/puppet@production] monitoring: check graphite2003 metrics

https://gerrit.wikimedia.org/r/731434

Stashbot added a comment.Oct 19 2021, 9:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T09:37:11Z] <godog> move graphite/statsd writes to graphite2003 - T247963

gerritbot added a comment.Oct 19 2021, 9:37 AM

Change 731433 merged by Filippo Giunchedi:

[operations/puppet@production] statsd: failover writes to graphite2003

https://gerrit.wikimedia.org/r/731433

gerritbot added a comment.Oct 19 2021, 9:37 AM

Change 731436 merged by Filippo Giunchedi:

[operations/dns@master] wmnet: move writes to graphite2003

https://gerrit.wikimedia.org/r/731436

gerritbot added a comment.Oct 19 2021, 10:01 AM

Change 731917 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/deployment-charts@master] mwdebug: add graphite2003 to network policies

https://gerrit.wikimedia.org/r/731917

gerritbot added a comment.Oct 19 2021, 10:03 AM

Change 731918 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/mediawiki-config@master] ProductionServices: use graphite2003 for statsd

https://gerrit.wikimedia.org/r/731918

gerritbot added a comment.Oct 19 2021, 10:16 AM

Change 731918 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices: use graphite2003 for statsd

https://gerrit.wikimedia.org/r/731918

Stashbot added a comment.Oct 19 2021, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T10:21:26Z] <oblivian@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s)

Stashbot added a comment.Oct 19 2021, 10:22 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T10:22:05Z] <godog> flip mw statsd traffic with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/731918 - T247963

Stashbot added a comment.Oct 19 2021, 10:22 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T10:22:38Z] <oblivian@deploy1002> Synchronized tests/WmfConfigServicesTest.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s)

Stashbot added a comment.Oct 19 2021, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T10:45:26Z] <godog> bounce superset on an-tool1010 to pick up statsd changes - T247963

Stashbot added a comment.Oct 19 2021, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T10:45:37Z] <godog> bounce navtiming on webperf1001 to pick up statsd changes - T247963

Stashbot added a comment.Oct 19 2021, 10:50 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T10:50:05Z] <godog> bounce superset on an-tool1005 to pick up statsd changes - T247963

fgiunchedi mentioned this in T175087: Create a navtiming processor for Prometheus.Oct 19 2021, 12:05 PM
fgiunchedi mentioned this in T293761: statsd and gunicorn metrics for superset.Oct 19 2021, 12:19 PM
gerritbot added a comment.Oct 20 2021, 10:17 AM

Change 732273 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: use standard recipe for all graphite hosts

https://gerrit.wikimedia.org/r/732273

fgiunchedi updated the task description. (Show Details)Oct 20 2021, 10:19 AM
gerritbot added a comment.Oct 20 2021, 12:06 PM

Change 732273 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: use standard recipe for all graphite hosts

https://gerrit.wikimedia.org/r/732273

gerritbot added a comment.Oct 21 2021, 6:38 AM

Change 731917 merged by jenkins-bot:

[operations/deployment-charts@master] mwdebug: fix statsd network policy

https://gerrit.wikimedia.org/r/731917

ops-monitoring-bot added a comment.Oct 21 2021, 7:56 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host graphite1004.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 21 2021, 8:24 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host graphite1004.eqiad.wmnet with OS bullseye completed:

  • graphite1004 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110210755_filippo_29322_graphite1004.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
fgiunchedi updated the task description. (Show Details)Oct 21 2021, 8:46 AM
fgiunchedi mentioned this in T138821: extend existing graphite whisper files retention to five years.Oct 22 2021, 12:39 PM
fgiunchedi added a subtask: T294220: Graphite query timeout from ExtensionDistributor.Oct 25 2021, 9:13 AM
gerritbot added a comment.Oct 25 2021, 10:24 AM

Change 734224 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: bump fetch_timeout

https://gerrit.wikimedia.org/r/734224

gerritbot added a comment.Oct 25 2021, 10:24 AM

Change 734225 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: set CLUSTER_SERVERS empty with no remote servers

https://gerrit.wikimedia.org/r/734225

fgiunchedi updated the task description. (Show Details)Oct 25 2021, 12:03 PM
gerritbot added a comment.Oct 25 2021, 2:23 PM

Change 734277 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] Revert \"discovery: move read traffic to graphite2003\"

https://gerrit.wikimedia.org/r/734277

gerritbot added a comment.Oct 25 2021, 2:25 PM

Change 734278 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Revert \"statsd: failover writes to graphite2003\"

https://gerrit.wikimedia.org/r/734278

gerritbot added a comment.Oct 25 2021, 2:25 PM

Change 734279 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Revert \"monitoring: check graphite2003 metrics\"

https://gerrit.wikimedia.org/r/734279

gerritbot added a comment.Oct 25 2021, 2:25 PM

Change 734280 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] Revert \"wmnet: move writes to graphite2003\"

https://gerrit.wikimedia.org/r/734280

gerritbot added a comment.Oct 25 2021, 2:26 PM

Change 734281 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/mediawiki-config@master] Revert \"ProductionServices: use graphite2003 for statsd\"

https://gerrit.wikimedia.org/r/734281

gerritbot added a comment.Oct 26 2021, 8:43 AM

Change 734225 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: set CLUSTER_SERVERS empty with no remote servers

https://gerrit.wikimedia.org/r/734225

gerritbot added a comment.Oct 26 2021, 9:26 AM

Change 734277 merged by Filippo Giunchedi:

[operations/dns@master] Revert \"discovery: move read traffic to graphite2003\"

https://gerrit.wikimedia.org/r/734277

Stashbot added a comment.Oct 26 2021, 9:27 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-26T09:27:13Z] <godog> move read traffic back to graphite1004 - T247963

gerritbot added a comment.Oct 26 2021, 9:29 AM

Change 734279 merged by Filippo Giunchedi:

[operations/puppet@production] Revert \"monitoring: check graphite2003 metrics\"

https://gerrit.wikimedia.org/r/734279

gerritbot added a comment.Oct 26 2021, 9:39 AM

Change 734278 merged by Filippo Giunchedi:

[operations/puppet@production] Revert \"statsd: failover writes to graphite2003\"

https://gerrit.wikimedia.org/r/734278

Stashbot added a comment.Oct 26 2021, 9:40 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-26T09:40:19Z] <godog> flip back write traffic to graphite1004 (all but mediawiki) - T247963

gerritbot added a comment.Oct 26 2021, 9:40 AM

Change 734280 merged by Filippo Giunchedi:

[operations/dns@master] Revert \"wmnet: move writes to graphite2003\"

https://gerrit.wikimedia.org/r/734280

Stashbot added a comment.Oct 26 2021, 9:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-26T09:47:13Z] <godog> bounce navtiming on webperf1001 to pick up statsd changes - T247963

Stashbot added a comment.Oct 26 2021, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-26T09:49:16Z] <godog> bounce superset on an-tool1010 to pick up statsd changes - T247963

Stashbot added a comment.Oct 26 2021, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-26T09:49:24Z] <godog> bounce superset on an-tool1005 to pick up statsd changes - T247963

gerritbot added a comment.Oct 26 2021, 10:01 AM

Change 734281 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert \"ProductionServices: use graphite2003 for statsd\"

https://gerrit.wikimedia.org/r/734281

fgiunchedi closed this task as Resolved.Oct 26 2021, 11:45 AM
fgiunchedi claimed this task.
fgiunchedi added a subscriber: Joe.

This is complete! Both graphite2003 and graphite1004 run with Bullseye, the failover documentation is up to date. Thanks @Joe for the assistance with mw config deploys.

gerritbot added a comment.Oct 29 2021, 6:47 AM

Change 734224 abandoned by Filippo Giunchedi:

[operations/puppet@production] graphite: bump fetch_timeout

Reason:

Not needed

https://gerrit.wikimedia.org/r/734224

lmata moved this task from In progress to Done on the SRE Observability (FY2021/2022-Q2) board.Oct 29 2021, 4:24 PM
Stashbot added a comment.Dec 9 2021, 3:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-12-09T03:37:10Z] <cwhite> bounce superset on an-tool1010 and 1005 to pick up statsd changes T247963

fgiunchedi closed subtask T294220: Graphite query timeout from ExtensionDistributor as Resolved.Feb 16 2022, 8:43 AM
Stashbot added a comment.Nov 30 2022, 9:30 AM

Mentioned in SAL (#wikimedia-operations) [2022-11-30T09:30:06Z] <godog> bounce superset on an-tool1010 to pick up statsd changes - T247963

Stashbot added a comment.Nov 30 2022, 9:32 AM

Mentioned in SAL (#wikimedia-operations) [2022-11-30T09:32:48Z] <godog> bounce superset on an-tool1005 to pick up statsd changes - T247963

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL