Page MenuHomePhabricator

ProbeDown (miscweb wikikube)
Closed, ResolvedPublic

Description

Common information

  • address: 10.2.1.70
  • alertname: ProbeDown
  • family: ip4
  • instance: miscweb2003:30443
  • job: probes/custom
  • prometheus: ops
  • severity: task
  • site: codfw
  • source: prometheus
  • team: serviceops-collab

Firing alerts











Event Timeline

Jelto renamed this task from ProbeDown to ProbeDown (miscweb wikikube).Dec 12 2023, 9:52 AM
Jelto claimed this task.
Jelto triaged this task as Medium priority.

This is (most probably) unrelated to miscweb and was a bigger wikikube outage in codfw. miscweb did not see a increased traffic on Kubernetes (the instance: miscweb2003:30443 is misleading and a misconfiguration in the blackbox check. We have to migrate the checks away from the puppet miscweb hosts).

There was also a page for another wikikube service wikifunctions.
And also mediawiki

So I'm closing this task. I'll add a followup to fix the blackbox exporter and move all Kubernetes checks away from https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/microsites/monitoring.pp.

I'll also add a followup task for serviceops to look at kubernetes2047, as this host caused the issues.