I agree that it would be nice to have push notifications for iOS and Android available as an option in addition to the paging system but I have some doubts/questions about the specific choice of Prowl.

I'm sure that others would prefer the opposite ;), to go to the source package instead to check the binaries generated by that source package that needs to be rebuilt and to find the hosts affected.

Search capability deployed!

Yes indeed, that's already in the plan for improvements. The problem here is that the logging is done by the framework and the framework doesn't have (as of now) a way to know for each cookbook which parameters are safe to be logged and which not.

@hashar this has nothing to do with Cumin but the local bash on the Cumin master.
If you use double quotes the bash interprets what's inside and replaces any variable with their value. If a variable is not defined it replaces it with empty string, so $foo will be replaced by empty string e Cumin will receive as parameter echo ''.

Use "df | awk '{ print \$6}'"

I've reset the mgmt card (see https://wikitech.wikimedia.org/wiki/Management_Interfaces#Reset_the_management_card ), wait that it rebooted, run ipmi-sensors that told me that the cache was outdated and needed a flush, run ipmi-sensors -f and we where good to go.
Sensors are working again, and Icinga is happy, resolving.

In addition io T220787#5106275, from the top of my head I think we need also:

• check if the DSA script we're using to alarm on HP raid ( modules/raid/files/dsa-check-hpssacli ) has been updated upstream (Debian) and update it or patch it and send the patch upstream (cc @faidon )
• adapt modules/raid/files/get-raid-status-hpssacli.sh to detect which executable is available and act accordingly, assuming they have the same options. If not we need to adapt the script to handle the two different exectuables.

In T219908#5101343, @crusnov wrote:

Okay the only question that seems open in my mind is how does the service map serial to fqdn?

In T219908#5095526, @crusnov wrote:

1. spicerack/cumin calls an end-point, say PUT https://<deployment>/ipxe/<serial> with DH URL and parameters

The log from today makes me thing that there is some sort of race-condition when we reload icinga (triggered by puppet usually) and the passive checks coming in from NSCA.

Since this task last update we've migrated Icinga to new hosts (jessie -> stretch) and slightly different version of Icinga. Resolving.

And when we do, can we also drop the package_updates custom fact?

In T219908#5083435, @crusnov wrote:

I suppose the conversation we need is:

• Where will this live?

@fgiunchedi what are your thoughts on T219854#5076968 ? That's the last remaining part of this task I guess.

Forgot to mention that during the reboot it printed:

Slot 3 Port 1 : Smart Array P840 Controller - (4096 MB, V3.56) 14 Logical
Drive(s) - Operation Failed
 - 1719-Slot 3 Drive Array - A controller failure event occurred prior
   to this power-up.  (Previous lock up code = 0x13) Action: Install the
   latest controller firmware. If the problem persists, replace the
   controller.

After the reboot the host is back up and running, all seems good so far. Keeping open for a bit to see if it holds.

So the dsa-check-hpssacli check is happily returning 0 exit code and this output:

OK: Slot 0: no logical drives --- Slot 0: no drives

Given that IIRC we add the HP raid check only on the hosts that have it, we might consider patching this imported script to fails in the case there is a controller but has no drives configured (both no logical and no physical?)

I can ssh into it via cumin.

@ema given the speedup due to prometheus 2 do you think this still needs to be worked on or could be resolved?

Check is live since a bit, contact list will be slowly grow, resolving.

It's all in the logs mentioned at the start of the script:

Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::openstack::codfw1dev::observer_password in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/openstack/codfw1dev/observerenv.pp:4:26 on node cloudcontrol2001-dev.wikimedia.org

It's kinda expected, the line when it says:

Scheduled delayed downtime on Icinga

spawn a subprocess that downtime the host on icinga with a delay, this is due to the fact that we use exported resources for Icinga checks and to downtime the host we need first that the puppetmaster compiles the catalog for the host and exports it into puppet db. Only after that we can run puppet on Icinga to gather the new host configuration and then downtime it.

I'd start with the RO user and see where we're going with the spicerack Ganeti module and when we start feeling blocked by this re-evaluate.
Having all RO operations done via the API and just the RW via ssh might also be an option as final solution if we have concerns for the security of the RW API user.

Cluster capacity it's already directly exposed by Ganeti, see https://wikitech.wikimedia.org/wiki/Ganeti#Listing_cluster_nodes

Given that py2 EOL is at the end of 2019, I'm not sure it's worth to spend our energies to make this check smart at this point.
An alternative proposal could at some point to migrate CI to test our Python files with Python 3.4 (jessie), 3.5 (stretch) and 3.7 (buster) [optionally 3.6 too but seems unnecessary in our current infrastructure].

@TheAnarcat yes we open the firewall on the PuppetDB hosts only from specific hosts (puppetmasters and cumin masters basically) and use the Puppet certificate for the HTTPS part.

They should be all ok. Feel free to re-open in case of issues.

Also worth mentioning that tracking the IDs for missing ones is not enough because if the last one fails we should know in advance how many are supposed to be there.

@fgiunchedi agree that this is a new issue, and we need to fix two different scripts to have an automatic task created for this:

First of all thanks a lot for the report and the patch @TheAnarcat.
The choice of hardcoding HTTPS as a protocol in the PuppetDB backend was done assuming that most likely Puppet, PuppetDB and Cumin would be installed on different hosts, and given the private nature of the data stored in PuppetDB, requiring HTTPS seems a natural choice. Of course if all of them are on the same host and listen only on localhost it doesn't really matter.

One thing that is missing are the physical devices that belongs to a cluster, see https://netbox.wikimedia.org/virtualization/clusters/3/

Given that this is quite old I'm closing it as duplicate of T210723 that has a more recent discussion of possible solutions. (CC @colewhite )

So, I tried upstream opening https://bugs.python.org/issue36284 but it got closed because 3.4 and 3.5 are security fix only at this point.
I'll look into adding a workaround into Cumin itself but is not super trivial because the reload() does mess up a bit with existing things.

So, I was able to repro this on Python 3.4 and 3.5 but not on 3.6 and 3.7 where it works like a charm.

Ok, I was able to repro without any cumin involvment, I've created the following structure:

$ tree repro/
repro/
├── fail.py
└── __init__.py

This is quite weird an require some more in-depth analysis unfortunately.

@RobH Icinga runs on both hosts and generates the same load, being active or passive changes very little. I've been monitoring this host with our beta icinga meta-monitoring and so far so good. It has now 11 days of uptime and I have a proposal for tomorrow's Foundation's meeting to failback to icinga1001 as active server either this Thu. or next Mon. given that is seems stable now.

I've had a chat with @ayounsi about this.

@jcrespo that's T216832 and we were thinking to just create a home for the user (cc @MoritzMuehlenhoff )

My main concern here is that the concept of a single service owner is limited and doesn't reflect reality.
We have multiple roles for each server/service, not in all cases we have all those "roles" but I think it can be boiled down to:

Unless has a spare::system role, in that case it should be staged 😉

@Marostegui the different states and their transitions (when they are supposed to be updated) are described here:

• https://wikitech.wikimedia.org/wiki/Server_Lifecycle#States
• https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Server_transitions

@Cmjohnson my tests on icinga1001 are completed, so feel free to shutdown at will when parts are available.

Would that mean that the missing hours will be totally lost? In that case probably better to ask the users that were asking for longer retention to make sure we're not loosing any required data. (my 2 cents)

From the error reported in the upstream issue it seems that is data-dependent. Have you tried by any chance any other retention between 8500h and 10500h?
Can we enable any more debugging to get a better idea of which metric is throwing the error so that maybe we can just skip it instead?

There could be some throttling ongoing. Also from a very quick look at [1] we might be using an older API version/url...

FYI it crashed again:

--------------------------------------------------------------------------------
SeqNumber       = 481
Message ID      = SYS1003
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2019-02-22 04:04:12
Message         = System CPU Resetting.
FQDD            = iDRAC.Embedded.1#HostPowerCtrl
--------------------------------------------------------------------------------
SeqNumber       = 480
Message ID      = RAC0703
Category        = Audit
AgentID         = RACLOG
Severity        = Information
Timestamp       = 2019-02-22 04:04:10
Message         = Requested system hardreset.
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------

I've merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/492202 to fix the configuration and forced a puppet run on A:ganeti as ferm failed on all of them

@RobH and it crashed again already! I'll leave it down in case @Cmjohnson wants to attach a physical console.
Anyway, it's all yours, can be shutdown/reboot at will.

Hardware logs;

icinga1001 is unresponsive this morning (no ping, no ssh, black console), re-opening

yeah, I think they are a bit overwhelmed by the activity on GitHub, all the issues, etc. In the contributing page they state:

Due to an excessive backlog of feature requests, we are not currently accepting any proposals which substantially extend NetBox's functionality beyond its current feature set

I was made aware today of the existence of a wmf-utils repository that might have been created with that in mind but doesn't seem very used:
https://gerrit.wikimedia.org/g/wmf-utils/+/refs/heads/master

Any insight on upstream plans for reports based on open issues or their mailing list?
Because we can surely go around it for some specific cases like the sort, but if we want to expand the usage of reports those kind of "hacks" will not be enough.

It seems all good from megacli:

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
=== RaidStatus completed

It seems that one disk if failed in a way that is not even reported by megacli. The new version of the script reports:

=== RaidStatus (does not include components in optimal state)
name: Adapter #0

It seems that also PD: 8 is failed now:

			PD: 8 Information
			Enclosure Device ID: 32
			Slot Number: 8
			Drive's position: DiskGroup: 0, Span: 0, Arm: 8
			Media Error Count: 0
			Other Error Count: 110
			Predictive Failure Count: 0
			Last Predictive Failure Event Seq Number: 0