@cmooney, this should be fixed now. As of 20:20 UTC today, /var/lib/prometheus/node.d/ferm-mss.prom's contents on clouddumps1002 were:

# HELP ferm_mss_cfg MSS values for Ferm-based hosts
# TYPE ferm_mss_cfg gauge
ferm_mss_cfg{endpoint="208.80.154.242:2049",interface="ens1f0np0",protocol="IPv4"} 1440.0
ferm_mss_cfg{endpoint="208.80.154.242:443",interface="ens1f0np0",protocol="IPv4"} 1440.0
ferm_mss_cfg{endpoint="208.80.154.242:873",interface="ens1f0np0",protocol="IPv4"} 1440.0
ferm_mss_cfg{endpoint="208.80.154.242:2049",interface="lo",protocol="IPv4"} 1440.0
ferm_mss_cfg{endpoint="208.80.154.242:443",interface="lo",protocol="IPv4"} 1440.0
ferm_mss_cfg{endpoint="208.80.154.242:873",interface="lo",protocol="IPv4"} 1440.0
ferm_mss_cfg{endpoint="[2620:0:861:ed1a::3:242]:2049",interface="ens1f0np0",protocol="IPv6"} 1400.0
ferm_mss_cfg{endpoint="[2620:0:861:ed1a::3:242]:443",interface="ens1f0np0",protocol="IPv6"} 1400.0
ferm_mss_cfg{endpoint="[2620:0:861:ed1a::3:242]:873",interface="ens1f0np0",protocol="IPv6"} 1400.0
ferm_mss_cfg{endpoint="[2620:0:861:ed1a::3:242]:2049",interface="lo",protocol="IPv6"} 1400.0
ferm_mss_cfg{endpoint="[2620:0:861:ed1a::3:242]:443",interface="lo",protocol="IPv6"} 1400.0
ferm_mss_cfg{endpoint="[2620:0:861:ed1a::3:242]:873",interface="lo",protocol="IPv6"} 1400.0

@elukey: yep, feel free to close it! Thank you so much for helping us out.

I wasn't able to reproduce this on clouddumps1002:

@LSobanski another reimage attempt today failed for the same reason

I just updated the DNS records, @bcampbell. Let me know if there's any unexpected behavior or if I can close the ticket.

(proposed) runbook is here: https://wikitech.wikimedia.org/wiki/Traffic/Runbooks/CPHostDepooled
I know it's sparse, but I had a hard time thinking of and trying to find ways that someone outside of Traffic could troubleshoot this. I'm open to any and all suggestions for improvement.

I just updated our DNS records, @bcampbell. Let me know if there's any unexpected behavior or if I can close the ticket.

How (assuming it's even possible) can we switch to the DigitalOcean runner? I tried adding the cloud tag to the build stage and even overriding the build: block in .gitlab-ci.yml, with no success.

Sorry, I was (blindly) following the instructions on wikitech and didn't stop to think. I'll take care of this myself!

Thanks for the clarification. That sounds good, @MoritzMuehlenhoff. I have no objections to implementing this.

@MoritzMuehlenhoff I did some digging into what implementing this would entail. There are a couple of concerns, I think:

@ECohen_WMDE, how do you want to do the public key confirmation? One of the most common methods is to put your pubkey on your Mediawiki page, usually in the Contact section. You could also submit a Gerrit patchset. Feel free to reach out to me here (preferred) or on Slack (Chris Dobbins) or IRC (ChrisDobbins901_) if you have any questions.

From what I can tell by checking WMF-NDA group membership and the NDA spreadsheet, an NDA hasn't yet been signed. But @KFrancis would know better than I.

@Johannes_Richter_WMDE you should be set up now. If there's something I missed, please reply and I'll fix it ASAP!

@KFrancis: could you add @Johannes_Richter_WMDE to the NDA spreadsheet? I wasn't able to find that username, so if I'm mistaken, my apologies. Thank you!

@KFrancis: could you add @mahmoud.abdelsattar.wmde to the NDA spreadsheet? I wasn't able to find that username, so if I'm mistaken, my apologies. Thank you!

@mahmoud.abdelsattar.wmde you should be all set up. If not, please let me know and I'll fix it ASAP!

@ssingh: that's right. I thought about this a bit over the weekend, and I think the easiest approach is going to be doing a clean install in a VM, grabbing the yaml config, and diffing it against our existing settings. If this seems reasonable, I was thinking we could start with Wikimedia DNS to validate the approach. Any thoughts?

Edit: never mind