Page MenuHomePhabricator
Create Task
Maniphest T299168

Upgrade OpenSearch
Closed, ResolvedPublic
Actions

Description

OpenSearch 1.2.4 when available (Expected Jan 18)
OpenSearch Dashboards 1.2.0
Logstash 7.16.3

  • test upgrade process (beta)
    • logstash + logstash-plugins deployment
    • transition to logstash-output-opensearch plugin
    • make a backup
    • smoketest phatality
  • run upgrade (eqiad)
  • backup codfw dashboards and restore on eqiad
  • promote eqiad to active
  • run upgrade (codfw)

Details

SubjectRepoBranchLines +/-
hiera: map logstash.wm.o to kibana7.eqiadoperations/puppetproduction+1 -1
prepare for logstash 7.16.3operations/software/logstash/pluginsmaster+12 -4
logstash: switch to opensearch output plugin on production logstashoperations/puppetproduction+4 -4
logstash: install logstash-plugins on logging logstash clustersoperations/puppetproduction+5 -0
builder: add opensearch1 pbuilder hooks for logstash-plugins updateoperations/puppetproduction+31 -0
beta-logs: use opensearch output pluginoperations/puppetproduction+4 -4
Customize query in gerrit

Event Timeline

colewhite created this task.Jan 13 2022, 8:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 13 2022, 8:15 PM
colewhite moved this task from Inbox to Up next on the SRE Observability (FY2021/2022-Q3) board.Jan 13 2022, 8:15 PM
colewhite updated the task description. (Show Details)Jan 13 2022, 8:20 PM
colewhite updated the task description. (Show Details)Jan 18 2022, 10:48 PM
gerritbot added a comment.Jan 18 2022, 11:20 PM

Change 755040 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: use opensearch output plugin

https://gerrit.wikimedia.org/r/755040

gerritbot added a project: Patch-For-Review.Jan 18 2022, 11:20 PM
gerritbot added a comment.Jan 18 2022, 11:23 PM

Change 755040 merged by Cwhite:

[operations/puppet@production] beta-logs: use opensearch output plugin

https://gerrit.wikimedia.org/r/755040

gerritbot added a comment.Jan 18 2022, 11:34 PM

Change 755041 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/software/logstash/plugins@master] prepare for logstash 7.16.3

https://gerrit.wikimedia.org/r/755041

gerritbot added a comment.Jan 18 2022, 11:39 PM

Change 755043 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] builder: add opensearch1 pbuilder hooks for logstash-plugins update

https://gerrit.wikimedia.org/r/755043

colewhite updated the task description. (Show Details)Jan 18 2022, 11:53 PM
gerritbot added a comment.Jan 20 2022, 10:22 PM

Change 755043 merged by Cwhite:

[operations/puppet@production] builder: add opensearch1 pbuilder hooks for logstash-plugins update

https://gerrit.wikimedia.org/r/755043

gerritbot added a comment.Jan 20 2022, 10:45 PM

Change 755811 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: install logstash-plugins on logging logstash clusters

https://gerrit.wikimedia.org/r/755811

gerritbot added a comment.Jan 20 2022, 10:51 PM

Change 755812 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: switch to opensearch output plugin on production logstash

https://gerrit.wikimedia.org/r/755812

gerritbot added a comment.Jan 21 2022, 9:00 PM

Change 755811 merged by Cwhite:

[operations/puppet@production] logstash: install logstash-plugins on logging logstash clusters

https://gerrit.wikimedia.org/r/755811

colewhite updated the task description. (Show Details)Jan 25 2022, 1:02 AM
Stashbot added a comment.Jan 25 2022, 8:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T20:17:32Z] <cwhite> begin transition to logstash output opensearch plugin T299168

gerritbot added a comment.Jan 25 2022, 8:17 PM

Change 755812 merged by Cwhite:

[operations/puppet@production] logstash: switch to opensearch output plugin on production logstash

https://gerrit.wikimedia.org/r/755812

Stashbot added a comment.Jan 25 2022, 9:03 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T21:03:07Z] <cwhite> end transition to logstash output opensearch plugin T299168

gerritbot added a comment.Jan 27 2022, 12:43 AM

Change 755041 merged by Cwhite:

[operations/software/logstash/plugins@master] prepare for logstash 7.16.3

https://gerrit.wikimedia.org/r/755041

Stashbot added a comment.Feb 1 2022, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-01T17:47:52Z] <cwhite> begin logstash upgrade (eqiad) T299168

Stashbot added a comment.Feb 1 2022, 6:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-01T18:10:07Z] <cwhite> end logstash upgrade (eqiad) T299168

Stashbot added a comment.Feb 2 2022, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-02T17:26:43Z] <cwhite> begin logstash upgrade (codfw) T299168

Stashbot added a comment.Feb 2 2022, 5:45 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-02T17:45:24Z] <cwhite> end logstash upgrade (codfw) T299168

colewhite added a comment.EditedFeb 2 2022, 6:11 PM

Upgrade on beta successful. Observed that plugins are not removed from opensearch and opensearch-dashboards on upgrade and will prevent the services from starting. Phatality (among others) must be removed prior to upgrade then a compatible version reinstalled post-upgrade. Also, the artifact generated during the deb build process is opensearch-min but when imported replaces opensearch.

colewhite updated the task description. (Show Details)Feb 2 2022, 6:12 PM
colewhite updated the task description. (Show Details)Feb 2 2022, 6:37 PM
gerritbot added a comment.Feb 4 2022, 8:37 PM

Change 759783 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: map logstash.wm.o to kibana7.eqiad

https://gerrit.wikimedia.org/r/759783

colewhite changed the task status from Open to In Progress.Feb 4 2022, 11:10 PM
colewhite triaged this task as Medium priority.
Stashbot added a comment.Feb 7 2022, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-07T22:25:43Z] <cwhite> begin opensearch upgrade (eqiad) T299168

Stashbot added a comment.Feb 7 2022, 11:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-07T23:17:03Z] <cwhite> end opensearch upgrade (eqiad) T299168

colewhite updated the task description. (Show Details)Feb 7 2022, 11:17 PM
colewhite updated the task description. (Show Details)
gerritbot added a comment.Feb 15 2022, 4:17 PM

Change 759783 merged by Cwhite:

[operations/puppet@production] hiera: map logstash.wm.o to kibana7.eqiad

https://gerrit.wikimedia.org/r/759783

colewhite updated the task description. (Show Details)Feb 15 2022, 4:39 PM
Stashbot added a comment.Feb 15 2022, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T16:39:32Z] <cwhite> logstash switchback to eqiad complete T299168

Stashbot added a comment.Feb 22 2022, 8:26 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-22T20:26:54Z] <cwhite> begin opensearch upgrade (codfw) T299168

Stashbot added a comment.Feb 22 2022, 9:19 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-22T21:19:36Z] <cwhite> end opensearch upgrade (codfw) T299168

colewhite closed this task as Resolved.Feb 22 2022, 9:20 PM
colewhite updated the task description. (Show Details)
lmata moved this task from Up next to Done on the SRE Observability (FY2021/2022-Q3) board.Apr 11 2022, 1:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL