All set!

In T414374#11792638, @VRiley-WMF wrote:

Under @Papaul guidence, I have doubled checked the PERC controller, I found that one of the cables became unseated. Could you please try this again?

Hey @VRiley-WMF I'm not able to log in. Can you check the user/password?

boxes are imaged, in replication, and ready for traffic once pfw policy is done

hosts are up and running

In T417295#11782255, @Jclark-ctr wrote:

@Jgreen replaced cable link came up. Sorry for delay

@Jclark-ctr can you check the network cables for fransw1002? The first network interface doesn't appear to have link.

Password has been set to frack mgmt password for all three.

passwords have been reset to the frack mgmt password

All four are switched to UEFI and built.

@VRiley-WMF I'm not having much luck with this box. Running into two more issues:

In T416249#11758413, @Jclark-ctr wrote:

In T416249#11755868, @Jgreen wrote:

@Jclark-ctr I finally had a chance to look into the frmx1002/frdata1003 issue. I think the servers are in the correct rack, but I think they were put in the wrong vlan, so they'll need port and IP changes. These two servers should be in frack-listenerdmz not frack-fundraising.

@cmooney Could you assist with this next week?

@Jgreen Is there a way to determine which VLAN should be assigned to each server? The available options I see are:

fundraising
administration
payments
listenerdmz
bastion

frqueue1005/frqueue1006 are up and running

@Jclark-ctr I finally had a chance to look into the frmx1002/frdata1003 issue. I think the servers are in the correct rack, but I think they were put in the wrong vlan, so they'll need port and IP changes. These two servers should be in frack-listenerdmz not frack-fundraising.

@VRiley-WMF I'm unable to log in with either the usual temporary password or the standard fundraising one. Can you please check it out?

frmx1002's management interface isn't accessible, doesn't respond to ping

Working on frqueue1005:

• disabled the "embedded" NICs
• set serial port address: COM2
• set console redirection after boot: enabled
• switched boot method to UEFI
• confirmed that first "integrated" NIC pxeboot is enabled

@Jclark-ctr we don't have the prod management password, only the frack one and a temporary one the other DC-Ops use for us. Can you reset these too?

@Jclark-ctr I just noticed it looks like these were configured in the frack-fundraising1-c-eqiad vlan, looks like I missed updating the install details when the task was created. Could you please move them to frack-payments1-c-eqiad and re-IP accordingly?

Note to self: since these are Supermicro, the default management user is "ADMIN"

These have all been updated to the frack management password.

In T418928#11670501, @RobH wrote:

Jeff,

I made assumptions on this since we didn't have racking details on the parent ordering task. Please doublecheck all racking details.

In T418628#11665628, @AKanji-WMF wrote:

In mid-sprint @Lars and @Eileenmcnaughton affirmed that we can deprecate.

In T416039#11656148, @Lars wrote:

@Jgreen This should keep our tmp table in civicrm under control in the future. I've deleted a bunch of old import jobs / tables from before this change.

done!

In T417941#11646872, @jhathaway wrote:

In T417941#11636764, @Dzahn wrote:

@Jgreen I removed the dmarc@donate.wikimedia.org line from that alias.

I did not find any "dmarc-ruf@" alias in that postfix alias file though.

It is being mentioned in T167337 and I guess it means this is in Google. But to be sure let me add @jhathaway

correct, we don't currently have an alias setup for dmarc-ruf, I'm going to remove it from our DNS record, since most providers don't actually send the reports, and there are privacy issues around the contents.

In T417941#11636764, @Dzahn wrote:

@Jgreen I removed the dmarc@donate.wikimedia.org line from that alias.

I did not find any "dmarc-ruf@" alias in that postfix alias file though.

It is being mentioned in T167337 and I guess it means this is in Google. But to be sure let me add @jhathaway

no longer necessary

In T417941#11634533, @Dzahn wrote:

There is indeed a postfix alias dmarc-rua@wikimedia.org in the private puppet repo.

But it forwards to 5 different addresses, both inside and outside of wikimedia.org

Do you just want that 1 line removed that says "dmarc@donate.wikimedia.org"?

This doesn't seem necessary or relevant since a lot of work has been done toward DMARC compliance for @wikimedia.org

We're using a third party service for this now.

This was reigned in around 2025-12-16 by adding pattern matches to the jmx config to control what is exported.

tidy{} added to clean up stray output config

There's an updated version of dump_database in puppet, which has a table exclusion feature. This has been deployed to frdb2005 and as of today we're archiving one backup from the old script (frdb1004) and one from the new script (frdb2005).

Normally the mariadb database for payments-wiki is readonly, with the read queries from each application server hitting the local replica. This is a security and availability feature, i.e. there is no dependency on the origin server being available except when the wiki is unlocked for edits. Changing to a read/write database model creates a new SPOF at payments1005.

Done!

@jgleeson this is deployed, it should start collecting once the 6th field is added

In T416525#11603695, @jgleeson wrote:

@Jgreen after one small tweak to your regex, I think we're good! https://regex101.com/r/XFOLHG/1

(?<value>[\d\.])+s? needed to be (?<value>[\d\.]+)s?

There should always be the same number of '|' delimiters. From the collection side it doesn't matter if there's a value between them. The regex is just looking for delimiters and capturing the values (including null) between them.

In T416251#11584083, @Jhancock.wm wrote:

@Dwisehaupt or @Jgreen can i rack this in the new rack? or is this going in the og one at codfw?

Added a prometheus log metric dns_timeout to track this per-host.

@herron TY!

Done!

• NIC.Embedded.1-1-1 pxe disabled
• NIC.Integrated.1-1-1 pxe enabled
• boot method changed from UEFI to BIOS
• fixed serial settings
  • Serial Communication: On with Console Redirection (was On without Console Redirection)
  • Serial Port Address: COM2 (was COM1)