Page MenuHomePhabricator
Create Task
Maniphest T414037

Consider enabling Blackbox K8s autodiscovery on dse-k8s Prometheus instance
Closed, ResolvedPublic
Actions

Description

Per parent ticket, we are need a way to monitor the TLS certificate lifetime for our OpenSearch on K8s platform. Also, because OpenSearch terminates TLS directly instead of going through Envoy, we might not be able to get the same kind of HTTP metrics, error codes, etc that we can typically get for k8s-hosted services.†

We should be able to do this via Prometheus' support for kubernetes_sd_config , which we already use to autodiscovery scrape targets hosted in K8s. There wouldn't be too much difference, but we'd need to a few things to target the Blackbox exporter instead of scraping metrics, see this article for examples.

Creating this ticket to:

  • Discuss our options with Observability.
  • Implement a TLS and HTTP health monitoring solution based on their recommendations.

† to be determined, the Envoy Telemetry (k8s) dashboard wasn't working for me when I wrote this.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
opensearch-ipoid: move service to "production" status.operations/puppetproduction+1 -1
[DO NOT MERGE] opensearch-ipoid: Add a path and timeout to blackbox checkoperations/puppetproduction+2 -0
Revert "Move opensearch-ipoid to production state"operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Jan 7 2026, 11:00 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJan 7 2026, 11:00 PM
Gehel edited projects, added Data-Platform-SRE (2026.01.05 - 2026.01.23); removed Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:15 PM
tappof added a comment.Jan 8 2026, 4:59 PM

If you need to monitor endpoints and/or TLS certificates, it might be sufficient to add the probes key under the service in the service catalog:

service::catalog:
  apertium:
    description: Machine Translation service. apertium.discovery.wmnet
    ...
    page: false
    probes:
      - type: http
        path: /listPairs
    ...
gerritbot added a comment.Jan 8 2026, 11:00 PM

Change #1224827 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] Revert "Move opensearch-ipoid to production state"

https://gerrit.wikimedia.org/r/1224827

gerritbot added a project: Patch-For-Review.Jan 8 2026, 11:00 PM

Change #1224827 merged by Ryan Kemper:

[operations/puppet@production] Revert "Move opensearch-ipoid to production state"

https://gerrit.wikimedia.org/r/1224827

gerritbot added a comment.Jan 9 2026, 5:33 PM

Change #1224999 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] opensearch-ipoid: move service to "production" status.

https://gerrit.wikimedia.org/r/1224999

gerritbot added a comment.Jan 9 2026, 8:59 PM

Change #1224999 merged by Bking:

[operations/puppet@production] opensearch-ipoid: move service to "production" status.

https://gerrit.wikimedia.org/r/1224999

gerritbot added a comment.Jan 9 2026, 10:00 PM

Change #1225029 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] [DO NOT MERGE] opensearch-ipoid: Add a path and timeout to blackbox check

https://gerrit.wikimedia.org/r/1225029

gerritbot added a comment.Jan 9 2026, 10:11 PM

Change #1225029 abandoned by Bking:

[operations/puppet@production] [DO NOT MERGE] opensearch-ipoid: Add a path and timeout to blackbox check

Reason:

Not needed, scrape config is already present

https://gerrit.wikimedia.org/r/1225029

bking closed this task as Resolved.Jan 9 2026, 11:43 PM
bking moved this task from Backlog - project to Done on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.

Thanks for the suggestion, @tappof ! I can confirm that the current solution is working.

There's a small, non-urgent problem I wanted to mention: the current integration creates the scrape config on the ops prometheus instance (as opposed to the dse-k8s prometheus instance). This means that mainline SRE will be alerted, whereas we'd prefer for my team (Data Platform SRE) to be alerted. That being said, there are pre-existing tickets (T303744 and T398073) on the topic of improved targeting of alerts in k8s, so I don't think we need to rehash that here.

As such, I'll go ahead and close this out. Thanks again for your help!

bking updated the task description. (Show Details)Jan 9 2026, 11:44 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.Fri, Jan 16, 1:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits