Page MenuHomePhabricator

Deploy OpenSearch for Beta following production observability configurations
Open, Needs TriagePublic

Event Timeline

Change 711741 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: improve kafka_shipper rsyslog output ssl options

https://gerrit.wikimedia.org/r/711741

Change 713701 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] aptrepo: add opensearch 1.x component

https://gerrit.wikimedia.org/r/713701

Change 713701 merged by Cwhite:

[operations/puppet@production] aptrepo: add opensearch 1.x component

https://gerrit.wikimedia.org/r/713701

Following up from yesterday's conversations re: ssl and rsyslog, AFAICS the librdkafka options to disable verification are:

  • ssl.endpoint.identification.algorithm
  • enable.ssl.certificate.verification

So in rsyslog configuration sth like this:

action(type="omkafka"
       broker=["pontoon-kafka-01.monitoring.eqiad1.wikimedia.cloud:9093"]
       topic="kafka_topic"
       dynatopic="on"
       dynatopic.cachesize="1000"
       partitions.auto="on"
       template="syslog_json"
       queue.type="LinkedList" queue.size="10000" queue.filename="output_kafka_json"
       queue.highWatermark="7000" queue.lowWatermark="6000"
       queue.checkpointInterval="5"
       confParam=[ "security.protocol=ssl",
                   "ssl.ca.location=/etc/ssl/certs/Puppet_Internal_CA.pem",
                   "enable.ssl.certificate.verification=false",
                   "ssl.endpoint.identification.algorithm=none",
                   "compression.codec=snappy",
                   "socket.timeout.ms=10000",
                   "socket.keepalive.enable=true",
                   "queue.buffering.max.ms=50",
                   "batch.num.messages=1000" ]
)

I haven't verified (hah) but I think only ssl.endpoint.identification.algorithm=none is needed to stop validating that the brokers present a verifiable cert

  • ssl.endpoint.identification.algorithm

Sep 10 15:39:05 deployment-mediawiki11 rsyslogd[28176]: error setting custom configuration parameter 'ssl.endpoint.identification.algorithm=none': No such configuration property: "ssl.endpoint.identification.algorithm" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

  • enable.ssl.certificate.verification

Sep 10 15:40:21 deployment-mediawiki11 rsyslogd[28335]: error setting custom configuration parameter 'enable.ssl.certificate.verification=false': No such configuration property: "enable.ssl.certificate.verification" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

It seems these options aren't available in librdkafka 0.9.3 (stretch) or 0.11.6 (buster), but they are available in 1.6.0 (bullseye). The options appear in librdkafka >= 1.1.0.

  • ssl.endpoint.identification.algorithm

Sep 10 15:39:05 deployment-mediawiki11 rsyslogd[28176]: error setting custom configuration parameter 'ssl.endpoint.identification.algorithm=none': No such configuration property: "ssl.endpoint.identification.algorithm" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

  • enable.ssl.certificate.verification

Sep 10 15:40:21 deployment-mediawiki11 rsyslogd[28335]: error setting custom configuration parameter 'enable.ssl.certificate.verification=false': No such configuration property: "enable.ssl.certificate.verification" [v8.1901.0 try https://www.rsyslog.com/e/1000 ]

It seems these options aren't available in librdkafka 0.9.3 (stretch) or 0.11.6 (buster), but they are available in 1.6.0 (bullseye). The options appear in librdkafka >= 1.1.0.

Thank you for checking, not really an option to disable validation at least until stretch and buster are around :(

Change 721359 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch: fork elasticsearch module into opensearch module

https://gerrit.wikimedia.org/r/721359

Change 721385 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch_dashboards: fork kibana module into opensearch_dashboards module

https://gerrit.wikimedia.org/r/721385

Change 721386 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] icinga: fork icinga::monitor::elasticsearch::base_checks

https://gerrit.wikimedia.org/r/721386

Change 721388 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork elasticsearch profile into opensearch::server

https://gerrit.wikimedia.org/r/721388

Change 721389 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork elasticsearch base_checks for opensearch

https://gerrit.wikimedia.org/r/721389

Change 721391 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork kibana profile into opensearch::dashboards

https://gerrit.wikimedia.org/r/721391

Change 721395 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: fork elasticsearch::logstash into opensearch::logstash

https://gerrit.wikimedia.org/r/721395

Change 721397 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] role: add logging::opensearch::collector role

https://gerrit.wikimedia.org/r/721397

Change 721400 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] role: add logging::opensearch::data role

https://gerrit.wikimedia.org/r/721400