All elk5 hardware has been decommed at this point.

In T253810#6185337, @fgiunchedi wrote:

I've PoC this with check_ipmi_sensor which supports checking SEL
...
The downside of this approach is potentially old SEL entries that we'll have to clear as they are surfaced on first deployment. Going forward, the SEL will need clearing for such errors to let the icinga alert actually clear. Since if we deploy this we'll be routinely clear the SEL on errors, I think it is important to log its entries elsewhere too and for that we can deploy freeipmi-ipmiseld which polls SEL and logs to syslog.

sure, sounds good to me!

+1 for option 2, I think that will be a more straightforward approach overall.

Hi @BTullis, sure, I've just added you to analytics-alerts and you should be receiving these emails now.

Hi @FGoodwin, your ldap account has been added to group wmf. I'll transition this to resolved now, but please don't hesitate to reopen if any followup is needed. Thanks!

Looks reasonable to me, and thanks much for writing the patch!

Key updated, but gerrit unable to update task due to policy. Resolving!

Verified face to face via a google meet session

Hi @tchin, your ldap account is now a member of the wmf group. I'll transition to resolved now but please don't hesitate to reopen if any follow-up is needed. Thanks!

In T285754#7187636, @Ottomata wrote:

@herron, so we should do step 1 and then help Ben do step 2?

In T285754#7187160, @BTullis wrote:

I also have an item on my checklist to say that I should be in the cn=ops LDAP group.

There are instructions on how I can add myself to that group, but only once I have sudo access.

Can anyone confirm this requirement? If so, can it be done on this ticket, or should I raise a new one?

Shell account has been created, and ldap account has been added to group wmf

Sure I'll go ahead and prep a patch. I may have missed it, but what realname should be used for btullis?

In T285754#7184198, @Ottomata wrote:

@razzi will take care of this, and I will follow up with SRE on enabling root access after the initial access is granted.

Hey @ArielGlenn, Since this has been idling in the access request queue for some time I'm going to untag SRE-Access-Requests for the time being. If any follow up is needed please do re-tag. Thanks!

Hi @tchin could you please coordinate obtaining a comment of approval on this task from your manager?

Hi @FGoodwin could you please coordinate obtaining a comment of approval on this task from your manager?

Hi @NRodriguez there are a couple steps to check off in order to move forward on this request. When you have a moment could you please...

That's interesting about the same behavior happening in the opposite direction with a disk add. I guess that makes some sense in a bug-ish kind of way -- network device being renumbered as a side-effect of changing the VM device layout. I was worried it would happen on the next reboot, but sounds like this should be stable unless we were to change the disk layout again. Feeling much better about leaving it as-is now.

The 150G secondary disk has been removed from the prometheus3001 VM.

+1! I'll plan deploy the patch above (now amended to 80G retention), move data and release the vdb device from prometheus3001 next week.

Prometheus disk usage (106G) on prometheus3001 is larger than what can comfortably fit alongside the OS on the /dev/vda (128G) so a 150G /dev/vdb was added as /srv.

In T281266#7107718, @ops-monitoring-bot wrote:

cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: logstash1020.eqiad.wmnet

• logstash1020.eqiad.wmnet (PASS)
  • Downtimed host on Icinga
  • Found physical host
  • Downtimed management interface on Icinga
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB

In T283013#7093631, @Majavah wrote:

Puppet manifests are trying to do that but that line is not present on deployment-logstash04 for some reason. Is that there on prod?

Hi @cmooney I've added your VO account to the GMT+1 "batphone" rotation just now. If you'd like to adjust that please feel free to ping observability any time. Thanks!

Will give this a 1w grace period before handing off for disk removal/wipe

Thanks for correcting the oversight on arclamp/xenon, TIL

Along with migrating these hosts to buster I've deployed an updated config to make mwlog more of a multi-datacenter service.

In T282019#7062851, @Volans wrote:

I'm not sure if hiding it completely is a great choice. We could look to see if we could improve the check and distinguish between a missing host and other failures.

Some high level thoughts about how we might approach migrating:

Reimaging the eqiad kafka-logging hosts and configuring them with raid50 layout, this will give us 5T usable (as opposed to the current 3T raid10) per host.