On 2026-08-19 db2207 was the MariaDB master for the s2 cluster [1] in our codfw datacenter [2]. At ~02:13 UTC the db2207 host rebooted itself, causing the entire s2 cluster in codfw to enter a read-only state requiring immediate attention from a DBA.

This caused readers to experience replication lag [3] for ~31 minutes for the Wikis being served by s2 in codfw at the time of the incident, affecting the following Wikis: bgwiki, bgwiktionary, cswiki, enwikiquote, enwiktionary, eowiki, fiwiki, idwiki, itwiki, nlwiki, nowiki, plwiki, ptwiki, svwiki, thwiki, trwiki, zhwiki [4].

At ~02:37 UTC the DBA team started an emergency switchover to promote db2204 as the new s2 master in codfw [5], the switchover concluded at ~02:44 UTC [6] effectively resolving the incident.

We're still investigating the root cause that made the hsot to reboot itself but since this host has crashed in the past it was removed from being a candidate master [7][8] and monitoring notifications were disabled [9].

We opened a ticket with the hardware vendor (Dell) to aid in understanding the cause for the crash and will update once we figure it out.

[1] https://wikitech.wikimedia.org/wiki/MariaDB#Core_(MediaWiki_databases)
[2] https://wikitech.wikimedia.org/wiki/Codfw_data_center
[3] https://wikitech.wikimedia.org/wiki/MySQL#Replication_lag
[4] https://noc.wikimedia.org/conf/highlight.php?file=dblists/s2.dblist
[5] https://phabricator.wikimedia.org/T435270
[6] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1326967
[7] https://phabricator.wikimedia.org/T435279
[8] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1326979
[9] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327438

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	db2207: Disable notifications	Marostegui	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Jhancock.wm	T435271 db2207 down
Resolved	Marostegui	T435270 Switchover s2 master (db2207 -> db2204)
Resolved	Marostegui	T435279 Change s2 codfw candidate master
Open	None	T436078 Do not use masters/candidate masters that have Intel Xeon Gold 5317 CPU
Open	Marostegui	T436633 Move an eqiad DB host to x3 that doesn't have Intel Xeon 5317
Open	Marostegui	T436922 Switchover x1 master (db2191 -> db2249)
Open	Marostegui	T436923 Switchover x1 master (db1220 -> db1278)
Resolved	Marostegui	T437188 Switchover s5 master (db2213 -> db2157)

Event Timeline

corto created this task.Aug 19 2026, 2:39 AM

corto changed the task status from Open to In Progress.

corto triaged this task as High priority.

corto created this object with visibility "WMF-NDA (Project)".

corto created this object with edit policy "WMF-NDA (Project)".

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2026, 2:39 AM

Incident metadata

Google Doc

Eevans merged a task: T435269: db2207 down s2 master.Aug 19 2026, 2:40 AM

Eevans added a subscriber: andrea.denisse.

Marostegui added a subtask: T435270: Switchover s2 master (db2207 -> db2204).Aug 19 2026, 2:41 AM

Marostegui closed subtask T435270: Switchover s2 master (db2207 -> db2204) as Resolved.Aug 19 2026, 2:47 AM

@wiki_willy this host has crashed _again_ with the same symptoms: host rebooted itself and nothing on getsel
This is what getraclog shows though at the relevant times:

--------------------------------------------------------------------------------
SeqNumber       = 11954
Message ID      = USR0031
Category        = Audit
AgentID         = RACLOG
Severity        = Information
Timestamp       = 2026-08-19 02:32:17
Message         = Unable to log in for wmfroot from 208.80.153.110 using SSH.
Message Arg   1 = wmfroot
Message Arg   2 = 208.80.153.110
Message Arg   3 = SSH
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------
SeqNumber       = 11952
Message ID      = SYS1003
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2026-08-19 02:11:48
Message         = System CPU Resetting.
FQDD            = iDRAC.Embedded.1#HostPowerCtrl
--------------------------------------------------------------------------------
SeqNumber       = 11951
Message ID      = SYS1000
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2026-08-19 02:11:30
Message         = System is turning on.
FQDD            = iDRAC.Embedded.1#HostPowerCtrl
--------------------------------------------------------------------------------
SeqNumber       = 11950
Message ID      = SYS1001
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2026-08-19 02:11:02
Message         = System is turning off.
FQDD            = iDRAC.Embedded.1#HostPowerCtrl
--------------------------------------------------------------------------------
SeqNumber       = 11949
Message ID      = SYS1005
Category        = Audit
AgentID         = iDRAC
Severity        = Information
Timestamp       = 2026-08-19 02:11:02
Message         = The server power action is initiated because the host device initiated a power-down operation.
Message Arg   1 = the host device initiated a power-down operation
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------
SeqNumber       = 11948
Message ID      = LOG007
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2026-08-19 02:11:02
Message         = The previous log entry was repeated 1 times.
Message Arg   1 = 1
--------------------------------------------------------------------------------
SeqNumber       = 11947
Message ID      = SYS1003
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2026-08-19 02:10:46
Message         = System CPU Resetting.
FQDD            = iDRAC.Embedded.1#HostPowerCtrl
--------------------------------------------------------------------------------
SeqNumber       = 11946
Message ID      = SYS1005
Category        = Audit
AgentID         = iDRAC
Severity        = Information
Timestamp       = 2026-08-19 02:10:45
Message         = The server power action is initiated because the management controller initiated a hard-reset operation.
Message Arg   1 = the management controller initiated a hard-reset operation
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------
SeqNumber       = 11946
Message ID      = SYS1005
Category        = Audit
AgentID         = iDRAC
Severity        = Information
Timestamp       = 2026-08-19 02:10:45
Message         = The server power action is initiated because the management controller initiated a hard-reset operation.
Message Arg   1 = the management controller initiated a hard-reset operation
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------
SeqNumber       = 11945
Message ID      = RAC0703
Category        = Audit
AgentID         = RACLOG
Severity        = Information
Timestamp       = 2026-08-19 02:10:44
Message         = Requested system hardreset.
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------

Restricted Application added a project: DC-Ops. · View Herald TranscriptAug 19 2026, 2:54 AM

wiki_willy assigned this task to Jhancock.wm.Aug 19 2026, 3:05 AM

andrea.denisse updated the task description. (Show Details)Aug 19 2026, 3:39 AM

Marostegui moved this task from Triage to In progress on the DBA board.Aug 19 2026, 5:39 AM

Marostegui created subtask T435279: Change s2 codfw candidate master.

Marostegui closed subtask T435279: Change s2 codfw candidate master as Resolved.Aug 19 2026, 5:49 AM

@andrea.denisse I'm tentatively setting Severity 2, happy to discuss further if you disagree

@Marostegui for reference db2207 had a switchover (T434565) recently - 11 August - for the kernel update.

In T435271#12229855, @CWilliams-WMF wrote:

@Marostegui for reference db2207 had a switchover (T434565) recently - 11 August - for the kernel update.

Thanks - this host crashed before unfortunately so I've removed it from being a candidate master anymore until Dell figures why it crashes randomly.

Marostegui changed the visibility from "WMF-NDA (Project)" to "Public (No Login Required)".Aug 19 2026, 8:45 AM

Marostegui changed the edit policy from "WMF-NDA (Project)" to "All Users".

Restricted Application added a subscriber: Stang. · View Herald TranscriptAug 19 2026, 8:45 AM

Maintenance_bot added a project: SRE.Aug 19 2026, 9:29 AM

I have some updates and a quick ask.
ask first, is it okay for me to do some firmware updates on this server right now?

updates:

i pulled some info from the db servers in service. CPU, cpu count, bios versions, idrac versions. The vast majority of the db servers that have ever had a crash have two Intel Xeon Gold 5317 3G, 12C/24T for cpu, and 1.12.1 bios version. The few others had DIMM and CPU errors to go with them.
the newest generation of those servers, Config-E, no longer uses that CPU. We haven't had a crash ticket as of writing this on the new Config-Es (knock on wood)
since we know this server has crashed after a bios and idrac update, we know that those two alone is not enough to fix the issue.
db2209 has been updated with some smaller component firmware updates (NIC, backplace, etc) and has been put back in service. we won't know if this is a fix unless it crashes.

so i guess there's a second ask. Is there a benchmark (1 month, 3 months, longer?) for thinking that the component firmware updates resolved the issue? I do not know what would be a good test for this. db2207 so far is the only one that has re-crashed.

I am opening another ticket with Dell and will be referring back to the service tickets for this one and db2209 to see if they've come up with anything new.

also, should the masters be moved off servers that have Intel Xeon Gold 5317? I have a list in a sheet that should be searchable by "DB Crash Report"

@Jhancock.wm the server was already depooled, so I have extended the downtime by 1 day and stopped MariaDB for you with the normal steps that we do for a reboot:

sudo mysql -Bse "stop replica"
sudo systemctl stop mariadb
sudo umount /srv
sudo swapoff -a

I have added the smaller component firmware upgrades to this one. So now if this one or db2209 crashes we should know if that helps. I'll update as soon as i can about any updates from Dell.

The server remains depooled and in downtime, but I have started replication and it should have caught up again shortly.

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Aug 19 2026, 9:04 PM

Change #1327438 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2207: Disable notifications

https://gerrit.wikimedia.org/r/1327438

Change #1327438 merged by Marostegui:

[operations/puppet@production] db2207: Disable notifications

https://gerrit.wikimedia.org/r/1327438

Maintenance_bot removed a project: Patch-For-Review.Aug 20 2026, 8:32 AM

I've disabled notifications for a few days, in case we need further work on it.

In T435271#12233874, @Jhancock.wm wrote:

also, should the masters be moved off servers that have Intel Xeon Gold 5317? I have a list in a sheet that should be searchable by "DB Crash Report"

Would you share that with me to check whether that's doable or not? Thank you!

emailed you a link

@Jhancock.wm @wiki_willy is there a way to search hosts with config E with Intel Xeon Gold 5317 in netbox? I am not finding how to.

andrea.denisse updated the task description. (Show Details)Aug 25 2026, 5:13 AM

In T435271#12233861, @Jhancock.wm wrote:

I have some updates and a quick ask.
ask first, is it okay for me to do some firmware updates on this server right now?

updates:

i pulled some info from the db servers in service. CPU, cpu count, bios versions, idrac versions. The vast majority of the db servers that have ever had a crash have two Intel Xeon Gold 5317 3G, 12C/24T for cpu, and 1.12.1 bios version. The few others had DIMM and CPU errors to go with them.

the newest generation of those servers, Config-E, no longer uses that CPU. We haven't had a crash ticket as of writing this on the new Config-Es (knock on wood)

since we know this server has crashed after a bios and idrac update, we know that those two alone is not enough to fix the issue.

db2209 has been updated with some smaller component firmware updates (NIC, backplace, etc) and has been put back in service. we won't know if this is a fix unless it crashes.

so i guess there's a second ask. Is there a benchmark (1 month, 3 months, longer?) for thinking that the component firmware updates resolved the issue? I do not know what would be a good test for this. db2207 so far is the only one that has re-crashed.

db1228 re-crashed in less than two months after a first crash and after upgrading bios and firmware:
Initial crash: https://phabricator.wikimedia.org/T430934
Upgraded bios and firmware: https://phabricator.wikimedia.org/T430934#12082613 and comments below.
Second crash today: https://phabricator.wikimedia.org/T435891

andrea.denisse updated the task description. (Show Details)Aug 25 2026, 5:18 AM

andrea.denisse updated the task description. (Show Details)Aug 25 2026, 5:50 AM

andrea.denisse updated the task description. (Show Details)Aug 25 2026, 6:34 AM

In T435271#12229855, @CWilliams-WMF wrote:

@Marostegui for reference db2207 had a switchover (T434565) recently - 11 August - for the kernel update.

Making queries with the heartbeat metric from Prometheus I can see the switchover of August 11 but the host seems to also have been down on August 13.

Screenshot From 2026-08-25 00-58-28.png (1,570×990 px, 201 KB)

In T435271#12250063, @andrea.denisse wrote:

In T435271#12229855, @CWilliams-WMF wrote:

@Marostegui for reference db2207 had a switchover (T434565) recently - 11 August - for the kernel update.

Making queries with the heartbeat metric from Prometheus I can see the switchover of August 11 but the host seems to also have been down on August 13.

I am not sure what that comes from, but db2207 became master 11th Aug (T434565) and there is no trace of that host being down the 13th (we'd have noticed as it was a master)

Captura de pantalla 2026-08-25 a las 9.10.07.png (678×300 px, 92 KB)

In T435271#12250077, @Marostegui wrote:

In T435271#12250063, @andrea.denisse wrote:

In T435271#12229855, @CWilliams-WMF wrote:

@Marostegui for reference db2207 had a switchover (T434565) recently - 11 August - for the kernel update.

Making queries with the heartbeat metric from Prometheus I can see the switchover of August 11 but the host seems to also have been down on August 13.

I am not sure what that comes from, but db2207 became master 11th Aug (T434565) and there is no trace of that host being down the 13th (we'd have noticed as it was a master)

I'm using this query on Thanos two weeks behind 2026-08-19 13:45:26 UTC, I was trying to gather replication lag metrics and was tinkering with the heartbet, that's when I noticed it. Tho it may be unrelated to the db host itself and could be an issue with ingestion, maybe Thanos was saturated that day or didn't receive the metrics adequately for any reason...

clamp_min( mysql_heartbeat_now_timestamp_seconds{ instance=~"db2207.*" } - mysql_heartbeat_stored_timestamp_seconds{ instance=~"db2207.*" }, 0 )

@Marostegui the cpu info isn't in netbox. i had to go through our procurement docs to figure out what each order of db servers had what in it.

In T435271#12251965, @Jhancock.wm wrote:

@Marostegui the cpu info isn't in netbox. i had to go through our procurement docs to figure out what each order of db servers had what in it.

Where can I see that info? The spreadsheet you shared doesn't seem to have that list, or I'm not finding it. Thanks!

these servers have that cpu
db1206-1263
db2185-2247

In T435271#12251965, @Jhancock.wm wrote:

@Marostegui the cpu info isn't in netbox. i had to go through our procurement docs to figure out what each order of db servers had what in it.

Hi, I gathered that data from lscpu using cumin. I also think that we could write a script to gather such data and populate our Netbox information automagically. Let me know if you'd also like to have this data for eqiad or if there's other information that can aid in the issue, hope it helps:

| Host               | CPU                                      |
| db2153.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2154.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2155.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2156.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2157.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2158.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2159.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2160.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2161.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2162.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2163.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2164.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2165.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2166.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2167.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2168.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2169.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2170.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2171.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2172.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2173.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2174.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2175.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2176.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2177.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2178.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2179.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2180.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2181.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2182.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2183.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2184.codfw.wmnet | Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz |
| db2185.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2186.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2187.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2188.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2189.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2190.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2191.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2192.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2193.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2194.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2195.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2196.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2197.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2198.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2199.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2200.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2201.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2202.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2203.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2204.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2205.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2206.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2207.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2208.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2209.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2210.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2211.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2212.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2213.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2214.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2215.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2216.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2217.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2218.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2219.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2220.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2221.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2222.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2223.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2224.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2225.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2226.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2227.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2228.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2229.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2230.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2231.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2232.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2233.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2234.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2235.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2236.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2237.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2238.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2239.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2240.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2241.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2242.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2243.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2244.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2245.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2246.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2247.codfw.wmnet | Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz |
| db2248.codfw.wmnet | Intel(R) Xeon(R) Gold 5412U |
| db2249.codfw.wmnet | INTEL(R) XEON(R) GOLD 6542Y |
| db2250.codfw.wmnet | INTEL(R) XEON(R) GOLD 6542Y |
| db2251.codfw.wmnet | INTEL(R) XEON(R) GOLD 6542Y |
| db2252.codfw.wmnet | INTEL(R) XEON(R) GOLD 6542Y |
| db2253.codfw.wmnet | INTEL(R) XEON(R) GOLD 6542Y |
| db2901.codfw.wmnet | Intel Xeon E3-12xx v2 (Ivy Bridge) |
| db2902.codfw.wmnet | Intel Xeon E3-12xx v2 (Ivy Bridge) |
| db2903.codfw.wmnet | Intel Xeon E3-12xx v2 (Ivy Bridge) |

andrea.denisse mentioned this in T436028: Add missing fields like CPU info to Netbox data.Aug 25 2026, 10:15 PM

In T435271#12254240, @andrea.denisse wrote:

In T435271#12251965, @Jhancock.wm wrote:

@Marostegui the cpu info isn't in netbox. i had to go through our procurement docs to figure out what each order of db servers had what in it.

Hi, I gathered that data from lscpu using cumin. I also think that we could write a script to gather such data and populate our Netbox information automagically. Let me know if you'd also like to have this data for eqiad or if there's other information that can aid in the issue, hope it helps:

I extended this to include the eqiad hosts along with BIOS and microcode versions, I created a paste [1] for it because it's a big file with plenty of rows. I also created a task to track the automation of data gathering to populate the missing fields in Netbox [2]. I know it may sound weird to run commands on so many hosts but it was 2ops on each host one to gather lscpu info and the other one for BIOS and microcode data so it didn't put a heavy toll on the DB hosts, I ended up processing the data on my local machine, hope it's useful!!

[1] P96257
[2] T436028

This is very useful @andrea.denisse thanks a lot. I'll leave this for DCOps but I think we can definitely benefit from such info in netbox cc @wiki_willy

For the shake of this task, does it need to be open or can this be closed - we've sort of hijacked it with different comments already.
From my side what I know is:

We are coordinating with Dell via email all this
This host is depooled, but I can repool it again
DCOps doesn't need to work on this host as of yet, they may if Dell asks for more info.

@Jhancock.wm thoughts?

Marostegui created subtask T436078: Do not use masters/candidate masters that have Intel Xeon Gold 5317 CPU.Aug 26 2026, 10:55 AM

I'm fine with closing this task.

also fwiw, there were no crash tickets for servers with the 5217 CPU, just 5317. i only mention that cause i saw that one pop up in andrea's pulled list. (nice btw!)

We'll be sure to let you know if Dell comes back to us to work on a server so that it can be safely taking out of production.

Jhancock.wm closed this task as Resolved.Aug 31 2026, 1:57 AM

Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 31 2026, 2:29 AM

Stang unsubscribed.Sep 3 2026, 8:33 PM

db2207 downClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Incident metadata

db2207 down
Closed, ResolvedPublic
Actions

Related Objects
Search...