[05:16:56] marostegui@cumin1003:~$ sudo db-switchover --skip-slave-move db1228 db1290
Starting preflight checks...
* Original read only values are as expected (master: read_only=False, slave: read_only=True)
* The host to fail over is a direct replica of the master
* Replication is up and running between the 2 hosts
* Binary log format is the same: MIXED
* The replication lag is acceptable: 0 (lower than the configured or default timeout)
Are you sure you want to switchover current master db1228 and promote db1290 instead [yes/no]? yes
Stopping heartbeat on db1228/(none)
Setting up original master as read-only
Slave caught up to the master after waiting 0.012254714965820312 seconds
Servers sync at master: db1228-bin.001908:102103606 slave: db1290-bin.000004:105009238
Stopping original master->slave replication
Setting up replica as read-write
All commands where successful, current status: original master read_only: 1 / original slave read_only: 0
Trying to invert replication direction
Starting heartbeat on db1290/(none)
Verifying everything went as expected...
SUCCESS: Master switch completed successfully
Enabling GTID on old master...
Updating zarcillo...
Traceback (most recent call last):
File "/usr/bin/db-switchover", line 33, in <module>
sys.exit(load_entry_point('wmfmariadbpy==0.12.1', 'console_scripts', 'db-switchover')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/wmfmariadbpy/cli_admin/switchover.py", line 775, in main
update_zarcillo(master, slave)
File "/usr/lib/python3/dist-packages/wmfmariadbpy/cli_admin/switchover.py", line 606, in update_zarcillo
result = zarcillo.execute(query.format(slave.host, slave.port, section, dc))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/wmfmariadbpy/WMFMariaDB.py", line 202, in execute
cursor.execute(command)
File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
result.read()
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
first_packet = self.connection._read_packet()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 725, in _read_packet
packet.raise_for_error()
File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/usr/lib/python3/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1048, "Column 'instance' cannot be null")Description
Details
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| switchover: detect if the replica is missing from zarcillo | repos/sre/wmfmariadbpy!27 | fceratto | T430104 | main | |
| db-switchover: add instance table safety check | repos/sre/wmfmariadbpy!26 | fceratto | T430104-instances-check | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T422365 Migration to Debian Trixie of production database-related hosts | |||
| Resolved | Marostegui | T423069 Migrate m2 to Debian Trixie | |||
| Resolved | Marostegui | T429929 Switchover m2 master (db1228 -> db1290) | |||
| In Progress | FCeratto-WMF | T430104 db-switchover failed when inserting into zarcillo |
Event Timeline
fceratto opened https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/merge_requests/26
db-switchover: add instance table safety check
I updated the PR. @Marostegui @CWilliams-WMF can we test it during a codfw switchover perhaps?
@FCeratto-WMF as per the MR, my tests showed multiple issues and perhaps others still to remedy; https://phabricator.wikimedia.org/P94650
fceratto opened https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/merge_requests/27
switchover: detect if the replica is missing from zarcillo
fceratto closed https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/merge_requests/27
switchover: detect if the replica is missing from zarcillo
We discussed in a meeting simplifying db-switchover but removing the SQL logic and delegating the work to the Zarcillo API; also log more clearly when the critical part of the switchover is done: SUCCESS: Master switch completed successfully
fceratto opened https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/merge_requests/28
switchover: Use Zarcillo API