Page MenuHomePhabricator
Create Task
Maniphest T288621

Logs and events produced by the WMF are consumed using the Elastic Common Schema by OpenSearch
Open, In Progress, MediumPublic
Actions

Description

This is a tracking task for OKR Work for this quarter:

  • T288618 - Deploy OpenSearch for Beta following production observability configurations
  • T288619 - Improve the process to consume and use API.LOG to filter out bad performing queries either by extra tooling within o11y or analytics
  • T288620 - Document path forward for how to Retire all non-Kafka Logstash inputs

Details

Other Assignee
herron
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

lmata created this task.Aug 11 2021, 1:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 11 2021, 1:16 PM
lmata added subtasks: T288618: Deploy OpenSearch for Beta following production observability configurations, T288619: Improve the process to consume and use API.LOG to filter out bad performing queries either by extra tooling within o11y or analytics, T288620: Document path forward and Retire remaining non-Kafka Logstash inputs.Aug 11 2021, 1:17 PM
lmata moved this task from Up next to Inbox on the SRE Observability (FY2021/2022-Q1) board.
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2021/2022-Q1) board.Aug 11 2021, 1:25 PM
lmata reassigned this task from lmata to colewhite.Aug 11 2021, 1:29 PM
lmata updated Other Assignee, added: lmata.
lmata updated Other Assignee, added: herron; removed: lmata.
colewhite edited projects, added SRE Observability (FY2021/2022-Q2); removed SRE Observability (FY2021/2022-Q1).Oct 1 2021, 12:17 AM
colewhite moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q2) board.Oct 1 2021, 12:20 AM
herron changed the status of subtask T288620: Document path forward and Retire remaining non-Kafka Logstash inputs from Open to In Progress.Oct 7 2021, 5:19 PM
colewhite closed subtask T288618: Deploy OpenSearch for Beta following production observability configurations as Resolved.Oct 27 2021, 9:59 PM
gerritbot added a comment.Nov 30 2021, 6:42 PM

Change 742778 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] site: consolidate logstash node definitions

https://gerrit.wikimedia.org/r/742778

gerritbot added a project: Patch-For-Review.Nov 30 2021, 6:42 PM

Change 742779 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] site: reprovision codfw logging cluster to opensearch

https://gerrit.wikimedia.org/r/742779

gerritbot added a comment.Nov 30 2021, 6:42 PM

Change 742780 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: add opensearch production configuration

https://gerrit.wikimedia.org/r/742780

gerritbot added a comment.Nov 30 2021, 6:42 PM

Change 742781 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch-dashboards: enable phatality on beta-logs

https://gerrit.wikimedia.org/r/742781

gerritbot added a comment.Nov 30 2021, 6:46 PM

Change 742781 merged by Cwhite:

[operations/puppet@production] opensearch-dashboards: enable phatality on beta-logs

https://gerrit.wikimedia.org/r/742781

gerritbot added a comment.Dec 1 2021, 9:07 PM

Change 742778 merged by Cwhite:

[operations/puppet@production] site: consolidate logstash node definitions

https://gerrit.wikimedia.org/r/742778

gerritbot added a comment.Dec 1 2021, 9:11 PM

Change 742780 merged by Cwhite:

[operations/puppet@production] hiera: add opensearch production configuration

https://gerrit.wikimedia.org/r/742780

gerritbot added a comment.Dec 1 2021, 11:41 PM

Change 743049 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] site: reprovision codfw logging cluster to opensearch

https://gerrit.wikimedia.org/r/743049

colewhite added a comment.EditedDec 3 2021, 6:05 PM

https://gerrit.wikimedia.org/r/743049

This is the last automated step in provisioning OpenSearch.

Before merging this change, Puppet should be disabled on the whole codfw ES cluster.

Optionally, disable shard allocation for the cluster. Between each node join, shard allocation will have to be re-enabled to allow the node to allocate and update its own shards.

After merge, the first nodes to migrate are data nodes. Do serially:

  • Stop elasticsearch
  • Enable and Run Puppet

After Puppet applies this change, ensure OpenSearch is stopped. Prior to joining the cluster, we'll want to put the ES index data into place:

  • mv /etc/elasticsearch/production-elk7-codfw /srv/opensearch/production-elk7-codfw
  • chown -R opensearch:opensearch /srv/opensearch/production-elk7-codfw Once data is in place, start OpenSearch and watch logs and api endpoints for a successful cluster join and shard provisioning.

After merge, the last nodes to migrate are collector nodes. Do serially:

  • Stop Logstash
  • Stop elasticsearch
  • Enable and Run Puppet

Some manual steps once complete:

  • Purge elasticsearch-oss and kibana packages
  • Disable and stop lingering services
    • sudo systemctl disable elasticsearch_7@production-elk7-codfw
    • sudo systemctl disable elasticsearch-production-elk7-codfw-gc-log-cleanup.timer
    • sudo systemctl stop elasticsearch-production-elk7-codfw-gc-log-cleanup.timer
  • Check for and possibly remove lingering files:
    • /etc/logrotate.d/elastic*
    • /etc/elasticsearch
    • /lib/systemd/system/elasticsearch*
    • /var/log/elasticsearch-production-elk7-codfw-gc-log-cleanup
    • /etc/kibana
    • /etc/default/kibana
    • /etc/sudoers.d/kibana-deploy-phatality

Lastly, check for functionality of OpenSearch Dashboards and ingest pipeline. Once determined functional, we'll restore a kibana backup and point the UI at codfw.

Stashbot added a comment.Dec 6 2021, 8:14 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-06T20:14:36Z] <cwhite> begin codfw opensearch upgrade T288621

gerritbot added a comment.Dec 6 2021, 8:20 PM

Change 743049 merged by Cwhite:

[operations/puppet@production] site: reprovision codfw logging cluster to opensearch

https://gerrit.wikimedia.org/r/743049

gerritbot added a comment.Dec 6 2021, 8:27 PM

Change 744088 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: synchronize cluster name

https://gerrit.wikimedia.org/r/744088

gerritbot added a comment.Dec 6 2021, 8:28 PM

Change 744088 merged by Cwhite:

[operations/puppet@production] hiera: synchronize cluster name

https://gerrit.wikimedia.org/r/744088

Stashbot added a comment.Dec 7 2021, 12:10 AM

Mentioned in SAL (#wikimedia-operations) [2021-12-07T00:10:18Z] <cwhite> end codfw opensearch upgrade T288621

gerritbot added a comment.Dec 7 2021, 6:58 PM

Change 744845 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch_dashboards: allow up to 64mb restore payload

https://gerrit.wikimedia.org/r/744845

herron closed subtask T288620: Document path forward and Retire remaining non-Kafka Logstash inputs as Resolved.Dec 7 2021, 8:39 PM
gerritbot added a comment.Dec 8 2021, 4:46 PM

Change 745284 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: map logstash.wm.o to kibana7.codfw

https://gerrit.wikimedia.org/r/745284

gerritbot added a comment.Dec 8 2021, 4:47 PM

Change 744845 merged by Cwhite:

[operations/puppet@production] opensearch_dashboards: allow up to 64mb restore payload

https://gerrit.wikimedia.org/r/744845

colewhite changed the task status from Open to In Progress.Dec 8 2021, 5:22 PM
colewhite triaged this task as Medium priority.
gerritbot added a comment.Dec 9 2021, 5:41 PM

Change 745284 merged by Cwhite:

[operations/puppet@production] hiera: map logstash.wm.o to kibana7.codfw

https://gerrit.wikimedia.org/r/745284

Stashbot added a comment.Dec 9 2021, 5:48 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-09T17:48:38Z] <cwhite> point kibana7 to OpenSearch in codfw T288621

gerritbot added a comment.Dec 10 2021, 7:18 PM

Change 742779 abandoned by Cwhite:

[operations/puppet@production] site: reprovision codfw logging cluster to opensearch

Reason:

https://gerrit.wikimedia.org/r/742779

herron updated the task description. (Show Details)Dec 15 2021, 2:35 AM
gerritbot added a comment.Jan 10 2022, 10:43 PM

Change 752755 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: add opensearch production configuration (eqiad)

https://gerrit.wikimedia.org/r/752755

gerritbot added a comment.Jan 10 2022, 10:43 PM

Change 752756 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] site: reprovision eqiad logging cluster to opensearch

https://gerrit.wikimedia.org/r/752756

gerritbot added a comment.Jan 11 2022, 11:29 PM

Change 752755 merged by Cwhite:

[operations/puppet@production] hiera: add opensearch production configuration (eqiad)

https://gerrit.wikimedia.org/r/752755

Stashbot added a comment.Jan 12 2022, 7:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-12T19:25:05Z] <cwhite> begin eqiad opensearch upgrade T288621

gerritbot added a comment.Jan 12 2022, 7:28 PM

Change 752756 merged by Cwhite:

[operations/puppet@production] site: reprovision eqiad logging cluster to opensearch

https://gerrit.wikimedia.org/r/752756

gerritbot added a comment.Jan 12 2022, 7:42 PM

Change 753547 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: fix opensearch common_settings namespace

https://gerrit.wikimedia.org/r/753547

gerritbot added a comment.Jan 12 2022, 7:45 PM

Change 753547 merged by Cwhite:

[operations/puppet@production] hiera: fix opensearch common_settings namespace

https://gerrit.wikimedia.org/r/753547

lmata edited projects, added SRE Observability (FY2021/2022-Q3); removed SRE Observability (FY2021/2022-Q2).Jan 12 2022, 9:43 PM
lmata moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q3) board.
Stashbot added a comment.Jan 12 2022, 10:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-12T22:48:04Z] <cwhite> end eqiad opensearch upgrade T288621

gerritbot added a comment.Jan 14 2022, 9:07 PM

Change 754035 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: allow logstash1032 through kafka-jumbo firewall

https://gerrit.wikimedia.org/r/754035

gerritbot added a comment.Jan 14 2022, 9:14 PM

Change 754035 merged by Cwhite:

[operations/puppet@production] hiera: allow logstash1032 through kafka-jumbo firewall

https://gerrit.wikimedia.org/r/754035

lmata updated the task description. (Show Details)Feb 14 2022, 3:20 AM
lmata mentioned this in T272238: Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence.Feb 14 2022, 4:16 AM
RhinosF1 added a subscriber: RhinosF1.Apr 10 2022, 4:48 PM
lmata edited projects, added SRE Observability (FY2021/2022-Q4); removed SRE Observability (FY2021/2022-Q3), Patch-For-Review, Goal.Apr 11 2022, 1:05 PM
colewhite edited projects, added SRE Observability (FY2022/2023-Q1); removed SRE Observability (FY2021/2022-Q4).Jul 1 2022, 10:58 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q2); removed SRE Observability (FY2022/2023-Q1).Nov 8 2022, 4:13 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q3); removed SRE Observability (FY2022/2023-Q2).Jan 12 2023, 6:51 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL