Mentioned in SAL (#wikimedia-releng) [2019-03-18T09:55:50Z] <hashar> deleting shutdowned instance integration-publisher02 , we do not use it anymore since doc publishing got overhauled ( T137890 ) # T218146

The configuration is generated via a python script modules/shinken/files/shinkengen in operations/puppet. It seems to query OpenStack for a list of instances on the project.

Turns out integration-publishing02 has been shutdown since it is no more used (T137890).

9 days after I have deleted the instance integration-publishing02, I still get notifications:

Notification Type: PROBLEM
Host: integration-publishing02
State: DOWN
Address: 172.16.4.5
Info: CRITICAL - Host Unreachable (172.16.4.5)

Date/Time: Wed 27 Mar 06:48:02 UTC 2019

@GTirloni apparently you have acted recently on shinken-02.shinken.eqiad.wmflabs instance. Would you mind checking the config? It seems to be stall at some arbitrary old point in time.

Supposedly puppet runs the script /usr/local/bin/shinkengen which get list of instances from openstack and generate the conf. It is executed on each puppet run unless /usr/local/bin/shinkengen --test-if-up-to-date and would notify the shinken service.

I notably still get notified for instances that no more exist :( integration-publishing02.integration.eqiad.wmflabs and integration-slave-docker-1046.integration.eqiad.wmflabs.

$ grep publishing02 /etc/shinken/generated/integration.cfg 
    host_name        integration-publishing02

shinkengen apparently stopped updating all projects except for tools on March 11th:

# ls -ltr
total 96
-rw-r--r-- 1 root    root    20860 Mar 11 14:57 deployment-prep.cfg
-rw-r--r-- 1 shinken shinken   711 Mar 11 14:57 extdist.cfg
-rw-r--r-- 1 shinken shinken  5370 Mar 11 14:57 analytics.cfg
-rw-r--r-- 1 shinken shinken  7303 Mar 11 14:57 integration.cfg
-rw-r--r-- 1 shinken shinken   244 Mar 11 14:57 shinken.cfg
-rw-r--r-- 1 shinken shinken   630 Mar 11 14:57 cvn.cfg
-rw-r--r-- 1 shinken shinken   255 Mar 11 14:57 codesearch.cfg
-rw-r--r-- 1 shinken shinken 39509 Mar 27 12:47 tools.cfg

The configuration seems okay:

# cat /etc/shinkengen.yaml 
projects: [ 'tools', 'deployment-prep', 'extdist', 'analytics', 'integration', 'shinken', 'cvn', 'codesearch' ]
output_path: '/etc/shinken/generated'
site: eqiad
keystone_host: cloudcontrol1003.wikimedia.org
keystone_port: 5000
puppet_enc_host: labs-puppetmaster.wikimedia.org

./modules/shinken/templates/shinkengen.yaml.erb uses the ::site variable to define the region and that doesn't match our region name anymore (the old OpenStack region was also named eqiad and everything was fine but now the new region is named eqiad1-r).

Change 499516 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] shinken: Convert to role/profile

https://gerrit.wikimedia.org/r/499516

As of just now, it looked to me like shinkengen was running properly but throwing an error due to a permissions problem. My guess is that 'shinkengen' was run as root and generated a new file that could not then be overwritten by the standard shinkengen run (which is prompted by puppet and runs as user 'shinken').

I wiped out the /etc/shinken/generated dir and started over and things seem happy now. This will probably also solve the zombie alerts, but time will tell.

Note, by the way, that in theory the multi-region issue was fixed with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494264/

In T218146#5061727, @GTirloni wrote:

./modules/shinken/templates/shinkengen.yaml.erb uses the ::site variable to define the region and that doesn't match our region name

I am pretty sure that the 'site' setting in shinkengen.yaml is only used to generate the fqdn, <site>.wmflabs. Since that's eqiad for both regions I think this is moot, although I could be overlooking another use of that var.

I think it is solved? At least I have not received ghost notifications this week.

It's still broken, I just received a host down alert for an instance I deleted two hours ago.

I was working on shinken just now, and it was broken/down for a couple of days due to adding a new project without a contact group. So I suspect today's warning wasn't evidence of shinkengen being broken.

Change 499516 merged by Andrew Bogott:
[operations/puppet@production] shinken: Convert to role/profile

https://gerrit.wikimedia.org/r/499516