Notes from today's deployment:

Added uid tburmeister to both wmf and wmf-nda. You should have access now, please feel free to reopen if there are any issues. Thanks!

Thanks for all the help @RobH! Marking this as resolved as the host is now pooled.

In T311264#8030866, @RobH wrote:

If it is just 'depool' from command line+stop puppet+icinga maint mode, I can handle so you don't need to take it down in advance of the work, just lemme know!

In T311264#8024288, @RobH wrote:

So if the idrac is accessible, the firmware update isn't OS impacting. However, I cannot login to this idrac interface via HTTPS or SSH, so it appears it'll have to be fully power drained to attempt to fix this issue. Is there anything preventing me from depooling this via command line and rebooting the host as needed?

@ssingh: ok for me to depool and reboot this host as needed via depool command in os?

In T311264#8023977, @Dzahn wrote:

We have had the "mgmt flapping"-issue in other DCs. In codfw a bunch of them were fixed after Papaul did firmware upgrades on the DRACs.

So I would suggest to check if you can get a firmware upgrade done in eqsin.

During the upgrade to bird2 today, the bird side of things seems to have caused no issues. The bird2 service started successfully and the configuration file was correct. However, we ran into an issue with the durum1001 host not advertising any prefixes and that led us to observing that anycast-healthchecker package was removed. This seems to have happened because anycast-healthchecker (henceforth called anycast-hc) depends on bird and we removed bird and installed bird2 as part of the upgrade.

In T310387#8003619, @Cmjohnson wrote:

The server was out of warranty, I swapped DIMM B1 with a DIMM from a spare. Server booted, no issues.

• Non-maintainer upload.
  • New upstream release 9.1.2

Hi @Papaul: Thanks for letting us know! The host is depooled and downtimed and so please proceed whenever you want. Thanks!

Thanks Moritz for the patch: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/790266. I tried this with doh6001 for T307427 and realized that we will need this to do a bit more since we do need to do a Puppet run to start all the relevant services.

In T305423#7901674, @Papaul wrote:

@ssingh the host is still has failed as status in netbox
https://netbox.wikimedia.org/dcim/devices/1611/

In T305423#7901638, @RobH wrote:

This host has had its ram replaced and booted into the OS successfully, detecting all memory without errors.

When we were replacing the memory, it forgot its bios time/date as the CR battery on the mainboard has discharged. This is only an issue if power is entirely lost, requiring the date/time to be set again. I'm not sure that its worth the downtime and such to swap out mainboard batteries when these are due for replacement in Q3 of next fiscal.

@ssingh: Would you be the one to return this host to service? If so, can you do so and resolve this task when done? Thank you!

For reference, depooling a Wikidough or durum host involves stopping the bird and bird6 services (after disabling Puppet but I think that's generic).

In T305423#7841765, @wiki_willy wrote:

Hi @ssingh - since this server is out of warranty and due to be refreshed in a few quarters, do you still want us to purchase a replacement DIMM to keep it up and running in the meantime or are you able to wait it out? Thanks, Willy

Thanks for the feedback @fgiunchedi and @Volans!

In T305589#7836863, @Dzahn wrote:

My 2 cents:

We have tested the above two changes with six cp host reboots and there are no concerns, confirming that this issue has been fixed.

In T303593#7796742, @gerritbot wrote:

Change 771610 merged by Ssingh:

[operations/puppet@production] P:icinga: add profile for performance tweaking

https://gerrit.wikimedia.org/r/771610

In T303724#7775296, @MusikAnimal wrote:

Thanks for reporting this! Understandably, the usernames are being removed from all revisions on this particular page. This is why XTools is breaking. I've got a fix coming in that will show limited data, and a warning explaining that because the usernames are missing, some data may be inaccurate or misleading.

IPv6 support for Wikidough and durum was finalized in T301165.

In T301165#7710616, @ayounsi wrote:

Puppet will automatically filter v4 from v6 neighbors and should do the right thing when v4 and v6 IPs are mixed in neighbors_list
https://github.com/wikimedia/puppet/blob/production/modules/bird/manifests/init.pp#L41

When no neighbors_list is set, it should also "do the right thing" by picking the v4 and v6 default's gateway:
https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/bird/anycast.pp#L26

So in theory the Bird side "should just works", but it has not been thoroughly tested.
What's for sure, is the v6 side on the routers is needed.

However I'm wondering if we should have a closer look at Bird 2 right now, before increasing our Bird 1 specific code.

In T301165#7694515, @cmooney wrote:

ssingh: thanks! Yeah I'm not aware of any reason not to just match what was done with the IPv4, even if there are other options in this case.

I've gone and added 3 IPs for Durum to mirror what was in place for the v4 allocation:

https://netbox.wikimedia.org/ipam/prefixes/515/ip-addresses/

Does that make sense? Let me know if there are any issues.

In T301165#7693858, @cmooney wrote:

We have discussed this in the Traffic team and decided to go with 2001:67c:930::1/128, mostly because we feel it's easy to memorize/copy (for cases where people want to do that and directly use the IP in their stub configuration).

My OCD will bite me for years but yes that is fine. Allocated in Netbox now:

https://netbox.wikimedia.org/ipam/prefixes/515/ip-addresses/

Not sure if you want to use the same /64 for durum or another one. Maybe we don't need to make a decision on that now.

We have discussed this in the Traffic team and decided to go with 2001:67c:930::1/128, mostly because we feel it's easy to memorize/copy (for cases where people want to do that and directly use the IP in their stub configuration).

doh1001 was also restarted but I forgot to add the -t switch and that's why you ops-bot didn't catch it :) Updated the hosts.

Mostly curious: is there anything that needs to be done for this as part of clinic duty?

Please don't close this task pending further discussion. Thank you.