Page MenuHomePhabricator

Differing database server ips and server_id numbers
Closed, ResolvedPublic

Description

server_id is calculated based on the ip, but some servers changed ip but did not change its server_id (puppet was not run before restart). Resulting on potential writes loss if they end up being part of the same replication cluster.

This:

while read host port; do diff -q <(../../mysql.py -BN --connect-timeout=3 -h $host -P $port -e "SELECT @@global.server_id") <(echo "print '$(dig +short $host)'.split('.').inject(0) {|total,value| (total << 8 ) + value.to_i}" | ruby; echo) || echo "$host $port is different"; done < hosts

Detected 2 cases, db1061 T195595 and db2045. Restart them.

Event Timeline

jcrespo created this task.May 25 2018, 3:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 25 2018, 3:46 PM
Marostegui updated the task description. (Show Details)May 25 2018, 3:51 PM
Marostegui moved this task from Triage to Next on the DBA board.May 25 2018, 4:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T05:20:50Z] <marostegui> Restart MySQL on db2045 (s8 codfw master) - T195598

Marostegui triaged this task as Medium priority.May 29 2018, 5:26 AM

db2045 has been restarted.
Before:

+------------+
| @@hostname |
+------------+
| db2045     |
+------------+
+-------------+
| @@server_id |
+-------------+
|   180363369 |
+-------------+

After:

+------------+
| @@hostname |
+------------+
| db2045     |
+------------+
+-------------+
| @@server_id |
+-------------+
|   180359242 |
+-------------+
Marostegui updated the task description. (Show Details)May 29 2018, 5:27 AM
Marostegui closed this task as Resolved.May 29 2018, 11:24 AM
Marostegui claimed this task.

The other host for this task, db1061 has been restarted. So the scope is done!

Vvjjkkii renamed this task from Differing database server ips and server_id numbers to 19baaaaaaa.Jul 1 2018, 1:07 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Marostegui as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
Marostegui renamed this task from 19baaaaaaa to Differing database server ips and server_id numbers.Jul 1 2018, 7:02 PM
Marostegui closed this task as Resolved.
Marostegui claimed this task.
Marostegui lowered the priority of this task from High to Medium.
Marostegui updated the task description. (Show Details)