FYI this has a new required dependency of Redis, so we should check with serviceops if we could use an existing Redis installation.
It also has some changes in the API and we should check if existing scripts needs to be updated.

FYI The current raid_handler.py could be adapted or (ideally) its generic parts extracted to be able to easily add other handlers for different types of checks. Both the state (WARNING, CRITICAL, etc..) and the state type (HARD, SOFT) can be passed to the handler that can decide what to do based on those.

I'm leaving it back to the current clinic duty (@fsero) at this point given that it needs to be re-worked, it's not just a follow-up.

Change is live:

L| 0 ~$ dig @ns0.wikimedia.org SRV _matrix._tcp.wikimedia.org

Both records are actually up now:

$ dig +trace wikimedia.modular.im
[...SNIP...]
wikimedia.modular.im.	300	IN	A	52.56.197.133
;; Received 65 bytes from 173.245.58.183#53(laura.ns.cloudflare.com) in 9 ms

I'm for option 2 as well, 3 breaks the contract that all facts are available at first puppet run and 1 doesn't really solve much in the long run, just the cron spam.

Bandaid applied, no output from running wmf-auto-restart on boron. It should be good for now. Leaving open for a better long-term solution.

Paste diff is not that smart, I just added a check for the main DYNA record to silently skip its CNAMEs without spamming stderr.

+1 on the naming and +1 on buster, they just have firewall rules, so should be pretty straightforward and easy to do.

I've asked @elukey to sync the account to HUE as I don't have access myself.

Added @Iflorez to the wmf LDAP group as agreed with @MoritzMuehlenhoff

Glad to hear, resolving then.

And with the above patch merged it should all be resolved. Reopen if needed.

As suggested by @dcausse let's try to capture a jstack next time it happens:

sudo -u gerrit2 jstack $(pidof java)

Restarted gerrit because was stuck and showed the same behaviour of the above graph:
https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=16&fullscreen&orgId=1&from=1559022591237&to=1559031673504

I've put the state of those hosts in Netbox back to active as they are currently "active" for the spare::system role and decomissioning should be set once we run the decom script (and it will be done automatically by the script very soon) and the host is removed from puppet completely.
I've also updated the documentation to reduce confusion:
https://wikitech.wikimedia.org/w/index.php?title=Server_Lifecycle&type=revision&diff=1827408&oldid=1827206

I've put the state of those hosts in Netbox back to active as they are currently "active" for the spare::system role and decomissioning should be set once we run the decom script (and it will be done automatically by the script very soon) and the host is removed from puppet completely.
I've also updated the documentation to reduce confusion:
https://wikitech.wikimedia.org/w/index.php?title=Server_Lifecycle&type=revision&diff=1827408&oldid=1827206

So after a bit of debugging with @mobrovac it seems that the alarm that is not notifying the team-services contact is the restbase endpoints health one that seems to be generated by the service::node Puppet define.
That define has support for adding custom contact groups, and I don't see any defined for the team-services in hieradata/.
According to @mobrovac this was working before so I'm wondering if anything changed recently elsewhere given that this part of the code hasn't AFAICT.

@mobrovac can you retry actions on the Icinga UI?

In T224406#5214485, @mobrovac wrote:

That still doesn't make sense. On that alias I do receive emails, so the alias part is ok. What is weird is that not all email notifs that used to come come any more. E.g. I get notifications about Cassandra being down on a node, but not about RESTBase. In the same vein, it seems I'm not receiving emails for SCB services either...

@mobrovac

• Regarding the notification AFAICT the RESTBase alerts notify the team-services group (services@). I don't see that alias defined in our exim configuration, so probably is managed by OIT. Could you check with them if that is still valid and includes the right people?
• Regarding the UI actions the user mobrovac is properly authorized in Icinga configuration, could you check that you're logged in with the correct case of the username?

I've added urbanecm to the LDAP group nda as per request above given that it's needed to check logstash during deployments.

As per docs added Urbanecm to the wmf-deployment group in Gerrit.

Forgot to mention, nothing in syslog or journalctl for MySQL on s2/s4 units.

Related documentation for the most useful messages:

cloudcontrol1003 is flapping its systemd degraded alert since 2019-05-25 21:46. The unit that fails is:

● designate_floating_ip_ptr_records_updater.service               loaded failed failed    Designate Floating IP PTR records updater

@fgiunchedi FYI we got some email to root@ from ms-be1014 with the following:

Cron <root@ms-be1014> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

@Nuria all the pre-requisites are there, is this approved?

All changes merged, confirmed that James can connect to few hosts.
As per docs added Jfishback to the wmf-deployment group in Gerrit.

@HMarcus change applied:

-legalquestions:	legal, liaison
+legalquestions:	legal

@Urbanecm is it ok to use the email you used to sign the NDA for the related patch in Puppet?
Keep in mind that that file is public.

@jrbs Any update on this?

Re-opening as the disk ended up in a failed state with 2 failed disks!
The automatic task was not opened because it was already in alarm in Icinga, so it didn't re-trigger. Here the status of megacli:

@HMarcus yes we have the current rule in the exim configuration:

legalquestions: legal, liaison

@Dzahn I guess @georgina would be more appropriate, the expiration contact should be related to the contractors point of contact, not the group's owner.

Given the latest updates removing Operations.

Checked with @Ottomata, no need for Operations here, removing the tag.

@RobH @faidon @crusnov: I've made the changes to the Lifecycle page, please have a look:
https://wikitech.wikimedia.org/w/index.php?title=Server_Lifecycle&type=revision&diff=1827206&oldid=1826423

@hashar another question for you. If I have 2 CRs, chained one on top of another and I +2 both of them because I want to deploy them together, and the first one fails but the second one passes, would the second one be rebased on top of production branch and merged despite the fact that its parent was not merged?
If this is the case this would be a blocker IMHO.

I got slightly nerd-sniped into this and had a look. First I found that the optimizer choose a full table scan even without joins, as soon as we select a field from the revision table that is not included in the index.

I've verified with @JFishback_WMF that basic access works as expected.

@darthmon_wmde could you verify all works as expected? Feel free to resolve this task if there isn't any problem.

In the meanwhile I've sent patches for the conversion to shell access and the analytics-privatedata-users inclusion.
Assigning to @greg for approval for the deployment group.

Pending approval from sponsor (@zeljkofilipin ) and deployment group owner (@greg )

I think it's fair to add transitions from pretty much any state to the failed state.

I'm assuming that in cases in which the rebase fails because of conflicts or the CI fails after the rebase Jenkins would vote -1 and the patch would be out of the merging queue. Correct me if I'm wrong.

Limit increased to 4096 and Icinga manually restarted on both 1001 and 2001.

@Thomas_Shafee I've created the mailing list as requested. All other options were left to their default.