Change 769928 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] certspotter: Temporarily disable certspotter

https://gerrit.wikimedia.org/r/769928

Change 769928 merged by Vgutierrez:

[operations/puppet@production] certspotter: Temporarily disable certspotter

https://gerrit.wikimedia.org/r/769928

Mentioned in SAL (#wikimedia-operations) [2022-03-11T10:25:34Z] <vgutierrez> disable certspotter - T303593

Change 770000 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] certspotter: add -start_at_end to only fetch new logs

https://gerrit.wikimedia.org/r/770000

Change 770000 merged by Ssingh:

[operations/puppet@production] certspotter: add -start_at_end to only fetch new logs

https://gerrit.wikimedia.org/r/770000

Change 770012 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] certspotter: re-enable systemd timer

https://gerrit.wikimedia.org/r/770012

Change 770012 merged by Ssingh:

[operations/puppet@production] certspotter: re-enable systemd timer

https://gerrit.wikimedia.org/r/770012

Change 771610 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:icinga: add profile for performance tweaking

https://gerrit.wikimedia.org/r/771610

Change 771610 merged by Ssingh:

[operations/puppet@production] P:icinga: add profile for performance tweaking

https://gerrit.wikimedia.org/r/771610

In T303593#7796742, @gerritbot wrote:

Change 771610 merged by Ssingh:

[operations/puppet@production] P:icinga: add profile for performance tweaking

https://gerrit.wikimedia.org/r/771610

https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=alert1001&var-datasource=thanos&var-cluster=alerting&from=now-6h&to=now

This has alleviated some of the network errors, so that's a good sign. (15:07 UTC mark).

I think we have two options to (try to) completely fix this issue:

• update certspotter to limit the concurrency when it fetches the logs. This does not mean reducing the CT log server it queries but to limit the concurrency so that it pauses between those queries.
• as per the commit message above, "We first start by setting interface::rps to the alerting_host role in order to improve the network performance. In the next commit, we will try to adjust the RX queuelen, which is currently set to 200 (with a maximum support of 2047)." We can try doing this manually to check if it makes a difference perhaps and then puppetize it.

No strong preferences on either one. We will be modifying certspotter to exclude our own certificates so that path has to be taken. The network fixes did help and that's quicker perhaps and worth the attempt.

In T303593#7797822, @ssingh wrote:

• as per the commit message above, "We first start by setting interface::rps to the alerting_host role in order to improve the network performance. In the next commit, we will try to adjust the RX queuelen, which is currently set to 200 (with a maximum support of 2047)." We can try doing this manually to check if it makes a difference perhaps and then puppetize it.

Sounds like a good next step to me 👍