Page MenuHomePhabricator
Create Task
Maniphest T288618

Deploy OpenSearch for Beta following production observability configurations
Closed, ResolvedPublic
Actions

Details

SubjectRepoBranchLines +/-
logstash: add opensearch output config definitionoperations/puppetproduction+69 -0
profile: logstash: add production logstash profileoperations/puppetproduction+290 -0
opensearch roles: apply profile::base classes according to realmoperations/puppetproduction+8 -2
hiera: add minimal logstash-beta-next hiera configurationoperations/puppetproduction+258 -0
profile: add beta logstash profileoperations/puppetproduction+182 -0
role: add logging::opensearch::collector roleoperations/puppetproduction+26 -0
role: add logging::opensearch::data roleoperations/puppetproduction+15 -0
profile: add logstash common profileoperations/puppetproduction+98 -0
profile: fork kibana profile into opensearch::dashboardsoperations/puppetproduction+156 -0
profile: improve kafka_shipper rsyslog output ssl optionsoperations/puppetproduction+83 -60
profile: fork elasticsearch profile into opensearch::serveroperations/puppetproduction+122 -0
profile: fork elasticsearch base_checks for opensearchoperations/puppetproduction+22 -0
profile: fork elasticsearch::logstash into opensearch::logstashoperations/puppetproduction+49 -0
logstash: kafka input: add manage_truststore parameteroperations/puppetproduction+6 -2
icinga: fork icinga::monitor::elasticsearch::base_checksoperations/puppetproduction+73 -0
opensearch_dashboards: fork kibana module into opensearch_dashboards moduleoperations/puppetproduction+122 -0
opensearch: fork elasticsearch module into opensearch moduleoperations/puppetproduction+1 K -0
aptrepo: add opensearch 1.x componentoperations/puppetproduction+2 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

lmata created this task.Aug 11 2021, 1:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 11 2021, 1:12 PM
lmata mentioned this in T288621: Logs and events produced by the WMF are consumed using the Elastic Common Schema by OpenSearch.Aug 11 2021, 1:16 PM
lmata added a parent task: T288621: Logs and events produced by the WMF are consumed using the Elastic Common Schema by OpenSearch.
gerritbot added a comment.Aug 11 2021, 10:18 PM

Change 711741 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: improve kafka_shipper rsyslog output ssl options

https://gerrit.wikimedia.org/r/711741

gerritbot added a project: Patch-For-Review.Aug 11 2021, 10:18 PM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2021/2022-Q1) board.Aug 18 2021, 4:42 PM
gerritbot added a comment.Aug 18 2021, 9:02 PM

Change 713701 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] aptrepo: add opensearch 1.x component

https://gerrit.wikimedia.org/r/713701

gerritbot added a comment.Aug 19 2021, 5:23 PM

Change 713701 merged by Cwhite:

[operations/puppet@production] aptrepo: add opensearch 1.x component

https://gerrit.wikimedia.org/r/713701

fgiunchedi subscribed.Sep 10 2021, 7:31 AM

Following up from yesterday's conversations re: ssl and rsyslog, AFAICS the librdkafka options to disable verification are:

  • ssl.endpoint.identification.algorithm
  • enable.ssl.certificate.verification

So in rsyslog configuration sth like this:

action(type="omkafka"
       broker=["pontoon-kafka-01.monitoring.eqiad1.wikimedia.cloud:9093"]
       topic="kafka_topic"
       dynatopic="on"
       dynatopic.cachesize="1000"
       partitions.auto="on"
       template="syslog_json"
       queue.type="LinkedList" queue.size="10000" queue.filename="output_kafka_json"
       queue.highWatermark="7000" queue.lowWatermark="6000"
       queue.checkpointInterval="5"
       confParam=[ "security.protocol=ssl",
                   "ssl.ca.location=/etc/ssl/certs/Puppet_Internal_CA.pem",
                   "enable.ssl.certificate.verification=false",
                   "ssl.endpoint.identification.algorithm=none",
                   "compression.codec=snappy",
                   "socket.timeout.ms=10000",
                   "socket.keepalive.enable=true",
                   "queue.buffering.max.ms=50",
                   "batch.num.messages=1000" ]
)

I haven't verified (hah) but I think only ssl.endpoint.identification.algorithm=none is needed to stop validating that the brokers present a verifiable cert

colewhite added a comment.Sep 10 2021, 4:53 PM
  • ssl.endpoint.identification.algorithm

Sep 10 15:39:05 deployment-mediawiki11 rsyslogd[28176]: error setting custom configuration parameter 'ssl.endpoint.identification.algorithm=none': No such configuration property: "ssl.endpoint.identification.algorithm" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

  • enable.ssl.certificate.verification

Sep 10 15:40:21 deployment-mediawiki11 rsyslogd[28335]: error setting custom configuration parameter 'enable.ssl.certificate.verification=false': No such configuration property: "enable.ssl.certificate.verification" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

It seems these options aren't available in librdkafka 0.9.3 (stretch) or 0.11.6 (buster), but they are available in 1.6.0 (bullseye). The options appear in librdkafka >= 1.1.0.

fgiunchedi added a comment.Sep 13 2021, 8:13 AM
In T288618#7345168, @colewhite wrote:
  • ssl.endpoint.identification.algorithm

Sep 10 15:39:05 deployment-mediawiki11 rsyslogd[28176]: error setting custom configuration parameter 'ssl.endpoint.identification.algorithm=none': No such configuration property: "ssl.endpoint.identification.algorithm" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

  • enable.ssl.certificate.verification

Sep 10 15:40:21 deployment-mediawiki11 rsyslogd[28335]: error setting custom configuration parameter 'enable.ssl.certificate.verification=false': No such configuration property: "enable.ssl.certificate.verification" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

It seems these options aren't available in librdkafka 0.9.3 (stretch) or 0.11.6 (buster), but they are available in 1.6.0 (bullseye). The options appear in librdkafka >= 1.1.0.

Thank you for checking, not really an option to disable validation at least until stretch and buster are around :(

gerritbot added a comment.Sep 15 2021, 5:23 PM

Change 721359 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch: fork elasticsearch module into opensearch module

https://gerrit.wikimedia.org/r/721359

gerritbot added a comment.Sep 15 2021, 7:32 PM

Change 721385 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch_dashboards: fork kibana module into opensearch_dashboards module

https://gerrit.wikimedia.org/r/721385

gerritbot added a comment.Sep 15 2021, 7:44 PM

Change 721386 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] icinga: fork icinga::monitor::elasticsearch::base_checks

https://gerrit.wikimedia.org/r/721386

gerritbot added a comment.Sep 15 2021, 7:55 PM

Change 721388 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork elasticsearch profile into opensearch::server

https://gerrit.wikimedia.org/r/721388

gerritbot added a comment.Sep 15 2021, 8:00 PM

Change 721389 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork elasticsearch base_checks for opensearch

https://gerrit.wikimedia.org/r/721389

gerritbot added a comment.Sep 15 2021, 8:44 PM

Change 721391 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork kibana profile into opensearch::dashboards

https://gerrit.wikimedia.org/r/721391

gerritbot added a comment.Sep 15 2021, 8:52 PM

Change 721395 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork elasticsearch::logstash into opensearch::logstash

https://gerrit.wikimedia.org/r/721395

gerritbot added a comment.Sep 15 2021, 8:56 PM

Change 721397 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] role: add logging::opensearch::collector role

https://gerrit.wikimedia.org/r/721397

gerritbot added a comment.Sep 15 2021, 9:09 PM

Change 721400 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] role: add logging::opensearch::data role

https://gerrit.wikimedia.org/r/721400

colewhite triaged this task as Medium priority.Sep 16 2021, 10:32 PM
gerritbot added a comment.Sep 24 2021, 10:36 PM

Change 723619 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: add minimal logstash-beta-next hiera configuration

https://gerrit.wikimedia.org/r/723619

colewhite edited projects, added SRE Observability (FY2021/2022-Q2); removed SRE Observability (FY2021/2022-Q1).Oct 1 2021, 12:17 AM
colewhite moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q2) board.Oct 1 2021, 12:20 AM
gerritbot added a comment.Oct 7 2021, 10:14 PM

Change 727624 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add opensearch output config definition

https://gerrit.wikimedia.org/r/727624

gerritbot added a comment.Oct 7 2021, 10:15 PM

Change 727625 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: kafka input: add manage_truststore parameter

https://gerrit.wikimedia.org/r/727625

gerritbot added a comment.Oct 7 2021, 10:15 PM

Change 727626 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: add logstash common profile

https://gerrit.wikimedia.org/r/727626

gerritbot added a comment.Oct 7 2021, 10:15 PM

Change 727627 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: add beta logstash profile

https://gerrit.wikimedia.org/r/727627

gerritbot added a comment.Oct 19 2021, 9:14 PM

Change 721359 merged by Cwhite:

[operations/puppet@production] opensearch: fork elasticsearch module into opensearch module

https://gerrit.wikimedia.org/r/721359

gerritbot added a comment.Oct 19 2021, 9:20 PM

Change 721385 merged by Cwhite:

[operations/puppet@production] opensearch_dashboards: fork kibana module into opensearch_dashboards module

https://gerrit.wikimedia.org/r/721385

gerritbot added a comment.Oct 19 2021, 10:21 PM

Change 721386 merged by Cwhite:

[operations/puppet@production] icinga: fork icinga::monitor::elasticsearch::base_checks

https://gerrit.wikimedia.org/r/721386

gerritbot added a comment.Oct 19 2021, 10:47 PM

Change 727625 merged by Cwhite:

[operations/puppet@production] logstash: kafka input: add manage_truststore parameter

https://gerrit.wikimedia.org/r/727625

gerritbot added a comment.Oct 19 2021, 11:39 PM

Change 721395 merged by Cwhite:

[operations/puppet@production] profile: fork elasticsearch::logstash into opensearch::logstash

https://gerrit.wikimedia.org/r/721395

gerritbot added a comment.Oct 19 2021, 11:51 PM

Change 721389 merged by Cwhite:

[operations/puppet@production] profile: fork elasticsearch base_checks for opensearch

https://gerrit.wikimedia.org/r/721389

gerritbot added a comment.Oct 20 2021, 8:14 PM

Change 721388 merged by Cwhite:

[operations/puppet@production] profile: fork elasticsearch profile into opensearch::server

https://gerrit.wikimedia.org/r/721388

gerritbot added a comment.Oct 20 2021, 8:59 PM

Change 732438 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: logstash: add production logstash profile

https://gerrit.wikimedia.org/r/732438

gerritbot added a comment.Oct 21 2021, 4:49 PM

Change 711741 abandoned by Cwhite:

[operations/puppet@production] profile: improve kafka_shipper rsyslog output ssl options

Reason:

https://gerrit.wikimedia.org/r/711741

gerritbot added a comment.Oct 21 2021, 11:20 PM

Change 721391 merged by Cwhite:

[operations/puppet@production] profile: fork kibana profile into opensearch::dashboards

https://gerrit.wikimedia.org/r/721391

gerritbot added a comment.Oct 22 2021, 9:41 PM

Change 727626 merged by Cwhite:

[operations/puppet@production] profile: add logstash common profile

https://gerrit.wikimedia.org/r/727626

gerritbot added a comment.Oct 22 2021, 9:52 PM

Change 721400 merged by Cwhite:

[operations/puppet@production] role: add logging::opensearch::data role

https://gerrit.wikimedia.org/r/721400

gerritbot added a comment.Oct 25 2021, 9:50 PM

Change 721397 merged by Cwhite:

[operations/puppet@production] role: add logging::opensearch::collector role

https://gerrit.wikimedia.org/r/721397

gerritbot added a comment.Oct 25 2021, 9:56 PM

Change 727627 merged by Cwhite:

[operations/puppet@production] profile: add beta logstash profile

https://gerrit.wikimedia.org/r/727627

gerritbot added a comment.Oct 25 2021, 10:18 PM

Change 723619 merged by Cwhite:

[operations/puppet@production] hiera: add minimal logstash-beta-next hiera configuration

https://gerrit.wikimedia.org/r/723619

gerritbot added a comment.Oct 26 2021, 3:23 PM

Change 734658 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch roles: apply profile::base classes according to realm

https://gerrit.wikimedia.org/r/734658

gerritbot added a comment.Oct 26 2021, 5:16 PM

Change 734658 abandoned by Cwhite:

[operations/puppet@production] opensearch roles: apply profile::base classes according to realm

Reason:

https://gerrit.wikimedia.org/r/734658

colewhite mentioned this in T285539: Easing pain points caused by divergence between cloudservices and production puppet usecases .Oct 27 2021, 3:56 PM
colewhite closed this task as Resolved.Oct 27 2021, 9:59 PM

New cluster MVP and announced.

colewhite mentioned this in T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.Oct 27 2021, 10:52 PM
lmata moved this task from In progress to Done on the SRE Observability (FY2021/2022-Q2) board.Oct 29 2021, 4:24 PM
hashar awarded a token.Nov 3 2021, 9:22 PM
gerritbot added a comment.Nov 24 2021, 10:54 PM

Change 732438 merged by Cwhite:

[operations/puppet@production] profile: logstash: add production logstash profile

https://gerrit.wikimedia.org/r/732438

gerritbot added a comment.Jan 18 2022, 10:45 PM

Change 727624 merged by Cwhite:

[operations/puppet@production] logstash: add opensearch output config definition

https://gerrit.wikimedia.org/r/727624

lmata mentioned this in T272238: Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence.Feb 14 2022, 4:16 AM
colewhite mentioned this in T360595: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations'.Mar 25 2024, 7:12 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 25 2024, 7:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL