Event Timeline
Ran into some puppet unpleasantness:
Could not find data item profile::ores::web::workers in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/ores/web.pp:4:20 on node deployment-ores01.deployment-prep.eqiad.wmflabs
@akosiaris I think this is caused by the situation you were describing, where the defaults aren't loaded on labs. Do you have any ideas why hiera wouldn't be picking up hieradata/role/common/ores.yaml? I've got role::ores enabled, not sure how else to encourage hiera here...
@awight you could either add the ores hiera values to the wiki page for that paject (ie i think deployment-prep) or add them through horizion.
@Paladox Good idea. For the record, I used the following values, and puppet seems healthy so far.
{
"profile::ores::celery::workers": 4,
"profile::ores::celery::queue_maxsize": 100,
"profile::ores::web::workers": 8,
"profile::ores::web::redis_host": "127.0.0.1"
}Puppet ran once, then broke itself:
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster02.deployment-prep.eqiad.wmflabs]
I've rebuilt the puppet certs.
Now there's a conflict between redis and prometheus-redis
It looks like this host has been rebuilt again:
$ ssh deployment-ores01 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:gNywH2BdkKg2mU45nnQhMo6HX336cntrTME3iKfbczo. Please contact your system administrator. Add correct host key in /home/alex/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/alex/.ssh/known_hosts:155 remove with: ssh-keygen -f "/home/alex/.ssh/known_hosts" -R deployment-ores01.deployment-prep.eqiad.wmflabs ECDSA host key for deployment-ores01.deployment-prep.eqiad.wmflabs has changed and you have requested strict checking. Host key verification failed.
Please don't reuse hostnames, the new one should've been deployment-ores02