Disable cirrus sanitizer job
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	RKemper
	Oct 30 2020, 11:01 PM

Description

cirrusCheckerJobs are/were building up in eqiad following the datacenter switchover. We need to figure out the source of the slowdown (perhaps SQL?), but for now disable the sane-itizer.

Some charts

https://grafana.wikimedia.org/d/VTCkm29Wz/envoy-telemetry?viewPanel=14&orgId=1&from=now-2d&to=now&var-datasource=eqiad%20prometheus%2Fops&var-origin=jobrunner&var-origin_instance=All&var-destination=cloudelastic-chi-https&var-destination=search-https_codfw&var-destination=search-https_eqiad shows the buildup in jobs

https://grafana.wikimedia.org/d/VTCkm29Wz/envoy-telemetry?viewPanel=14&orgId=1&from=1603806636038&to=1603962628850&var-datasource=eqiad%20prometheus%2Fops&var-origin=jobrunner&var-origin_instance=All&var-destination=cloudelastic-chi-https&var-destination=search-https_codfw&var-destination=search-https_eqiad shows that that requests to elastic didn't seem to be taking longer following the switchover - which means we should probably suspect the database

Details

	Subject	Repo	Branch	Lines +/-
	cirrus: temporarily disable saneitizer	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects

Mentioned Here: T266762: The saneitizer is a lot slower than when running in codfw (oct 27 2020 codfw -> eqiad switchover)

Event Timeline

RKemper created this task.Oct 30 2020, 11:01 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2020, 11:01 PM

Reedy added a project: Discovery-ARCHIVED.Oct 30 2020, 11:42 PM

Reedy updated the task description. (Show Details)

Change 637809 had a related patch set uploaded (by Dzahn; owner: Ryan Kemper):
[operations/puppet@production] cirrus: temporarily disable saneitizer

https://gerrit.wikimedia.org/r/637809

gerritbot added a project: Patch-For-Review.Oct 31 2020, 12:36 AM

Aklapper renamed this task from Disable cirrus saneitizer job to Disable cirrus sanitizer job.Oct 31 2020, 10:25 AM

• Hermann added a parent task: T267016: Plan for november meeting with Göteborgs museer.Nov 2 2020, 1:21 PM

DannyS712 removed a parent task: T267016: Plan for november meeting with Göteborgs museer.Nov 2 2020, 1:25 PM

RKemper claimed this task.Nov 5 2020, 5:31 AM

RKemper edited projects, added Discovery-Search (Current work); removed Discovery-ARCHIVED.

RKemper set the point value for this task to 2.

RKemper added a subscriber: Discovery-Search (Current work).

Change 637809 merged by Ryan Kemper:
[operations/puppet@production] cirrus: temporarily disable saneitizer

https://gerrit.wikimedia.org/r/637809

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2020, 5:10 AM

Aklapper removed a subscriber: Discovery-Search (Current work).Nov 9 2020, 10:45 AM

RKemper moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Nov 10 2020, 7:07 PM

Do we have a ticket to investigate and fix the underlying issue? And to re-enable saneitizer when done?

@Gehel https://phabricator.wikimedia.org/T266762 is the ticket to investigate the underlying issue. I added an AC in that ticket to re-enable saneitizer.

Gehel closed this task as Resolved.Nov 23 2020, 1:21 PM

Disable cirrus sanitizer jobClosed, ResolvedPublic2 Estimated Story PointsActions