Thanks a lot @elukey!

@Joe ack to all your replies, thanks for integrating the suggestions!

Quick first feedback/questions on the proposal:

This happened again today unfortunately. And because I don't see any logs of spurious passive checks from other frack hosts, I guess we have to discard the hypothesis that it might have been that the cause of the issue.

From a quick test the slowest one letter search was ~1s and was for less common letters like z or q. As of now I cannot repro the issue, feel free to resolve the task if you think that the new version have solved it too. It can be re-opened it case we found some repro.

@Cmjohnson which disk is a tricky question in this case.

While investigating the possible root causes for this I discovered that we had some new frack hosts just installed last week that were sending metrics although not yet fully configured. In particular they are not present in Icinga hostlist, hence Icinga discards those messages and in theory that shouldn't harm. But to avoid to have too many variables in place I've asked @Jgreen if it was possible to avoid sending the metrics at all until they are fully configured and he very kindly accepted and already implemented.

Having a look around in the system utility (ESC+9) I found that:

@jynus got it, thanks for the info. FYI if you want to test your workaround solution, there is another DB missing: frimpressions. I didn't re-create it though, as I have no context on it. I would have told you tomorrow ;)

@jcrespo FYI I was deploying debmonitor today and the replication broke on db1065 and db1117 because of missing debmonitor database.

I now see a SAL entry from @akosiaris:
11:18 akosiaris: powercycling ms-be1034, box is unresposive, tons of logs "sd 0:1:0:1: rejecting I/O to offline device"

Actually it seems that this already recovered: OK: Active: 4, Working: 4, Failed: 0, Spare: 0

@Cmjohnson is this controller really missing the battery or it's a software problem that is just not recognized?

Ack, I propose to leave it as is for now and re-evaluate once also Filippo is back. Resolving for now, feel free to re-open.

Forgot to mention that the above message and output was taken on labvirt1020 as I cannot ssh to 1019 right now.

I've double checked both the report script that populate this task and the Icinga check script that raised the alarm. The issue here seems to be that the controller in Slot 1 (the P840 actually used) doesn't have/recognize the battery, hence the CRITICAL:

I just discovered that this host is planned for reimage in the next few days, not bothering fixing the md array as the host is not seeing the replaced disk and might need anyway a reboot, going directly for the reimage at this point.

It looks to me that the battery is broken/not recognized.

@jcrespo naos has been reimaged to deploy2001.codfw.wmnet, so I guess it can now be added to the grants. Mentioning it here just for not forgetting, there is absolutely no hurry do to do it.

The proposed approach don't take into account hosts installed for the first time. As for detecting the newly added host on the Icinga configuration is not trivial at all, same for the disable notifications, that as of now requires a commit to hiera. Unless that part is moved to a more dynamic storage I don't see an easy fix going that path.

@akosiaris the current EDAC check is sum(increase($metric[4d])), so is checking the increase over the last 4 days, I'd say is not time-sensitive at all.

The CPU usage is already back to 40%, we can decide tomorrow if we want to increase the check_interval further.

To keep everyone in the loop, I've chat with @Catrope the other day about this and we debugged it a bit together.

For reference, last month CPU trend with the two clear increases:
https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?orgId=1&var-server=einsteinium&var-datasource=eqiad%20prometheus%2Fops&from=1524243342926&to=1526128864600&refresh=1m&panelId=3&fullscreen

Thanks @ArielGlenn for re-opening this. From a quick look we had two big increases, one on May 2nd and one on May 8th. I think they are related to those two changes that are basically adding a check for each host each:

• SMART failures: https://gerrit.wikimedia.org/r/#/c/427654/
• EDAC memory errors in the last 4d: https://gerrit.wikimedia.org/r/#/c/422110/

@Dzahn That usually happens if the alarm flap on icinga for some reason, the handler open a new task for each CRITICAL/HARD triggered by Icinga.

Yeah, puppetdb1001 will probably just generate some spam on IRC for failing puppet runs, transient.

Great! Thanks a lot.

As discussed in the monitoring meeting here some feedback:

@jcrespo ack, no blocker for me, I'm actually not using it.

In T192875#4156668, @jcrespo wrote:

@Volans you can speed up the process by setting some password on the private repository (and some non-private equivalent in the labs/private one), and suggesting a charset/collation for the database (utf8mb4?). Name debmonitor?

Setup DNS, DHCP, netboot and created 2 VMs on Ganeti: debmonitor[12]001.

I've an additional question, what is the expected behaviour in the following failure scenarios for each option?

I've downtimed db1098 on Icinga until Wed mid EU day and disabled notifications.

In T192875#4153775, @Marostegui wrote:

From where will you be querying this DB? (just to see which (new) grants you might need?

Sorry, I didn't mention the multi-DC setup :)

Personally never used, +1 to drop it.

To summarize the work done recently, I've made an audit of existing checks and fixed/improved some of them that had clear errors or needed to be updated. @chasemp has very kindly offered himself to review the WMCS related checks, users and groups.

@Joe no it would not be super easy to solve in a DRY way, I agree.

I've fixed it, it was a case of password misalignment, see one of the cases described in T150160,

Reporting it here too for the future, to fix it's sufficient to replace the --diff of the above command with --commit and then re-run the --diff to ensure that this time it will show no error.

Patch updated to overcome this problem, once reviewed and merged it should solve the issue.