Page MenuHomePhabricator

Differing database server ips and server_id numbers
Closed, ResolvedPublic

Description

server_id is calculated based on the ip, but some servers changed ip but did not change its server_id (puppet was not run before restart). Resulting on potential writes loss if they end up being part of the same replication cluster.

This:

while read host port; do diff -q <(../../mysql.py -BN --connect-timeout=3 -h $host -P $port -e "SELECT @@global.server_id") <(echo "print '$(dig +short $host)'.split('.').inject(0) {|total,value| (total << 8 ) + value.to_i}" | ruby; echo) || echo "$host $port is different"; done < hosts

Detected 2 cases, db1061 T195595 and db2045. Restart them.

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2018-05-29T05:20:50Z] <marostegui> Restart MySQL on db2045 (s8 codfw master) - T195598

Marostegui triaged this task as Medium priority.May 29 2018, 5:26 AM

db2045 has been restarted.
Before:

+------------+
| @@hostname |
+------------+
| db2045     |
+------------+
+-------------+
| @@server_id |
+-------------+
|   180363369 |
+-------------+

After:

+------------+
| @@hostname |
+------------+
| db2045     |
+------------+
+-------------+
| @@server_id |
+-------------+
|   180359242 |
+-------------+
Marostegui claimed this task.

The other host for this task, db1061 has been restarted. So the scope is done!

• Vvjjkkii renamed this task from Differing database server ips and server_id numbers to 19baaaaaaa.Jul 1 2018, 1:07 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Marostegui as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
Marostegui renamed this task from 19baaaaaaa to Differing database server ips and server_id numbers.Jul 1 2018, 7:02 PM
Marostegui closed this task as Resolved.
Marostegui claimed this task.
Marostegui lowered the priority of this task from High to Medium.
Marostegui updated the task description. (Show Details)