Page MenuHomePhabricator

db-switchover failed when inserting into zarcillo
Open, In Progress, HighPublic

Description

[05:16:56] marostegui@cumin1003:~$ sudo db-switchover --skip-slave-move db1228 db1290
Starting preflight checks...
* Original read only values are as expected (master: read_only=False, slave: read_only=True)
* The host to fail over is a direct replica of the master
* Replication is up and running between the 2 hosts
* Binary log format is the same: MIXED
* The replication lag is acceptable: 0 (lower than the configured or default timeout)
Are you sure you want to switchover current master db1228 and promote db1290 instead [yes/no]? yes
Stopping heartbeat on db1228/(none)
Setting up original master as read-only
Slave caught up to the master after waiting 0.012254714965820312 seconds
Servers sync at master: db1228-bin.001908:102103606 slave: db1290-bin.000004:105009238
Stopping original master->slave replication
Setting up replica as read-write
All commands where successful, current status: original master read_only: 1 / original slave read_only: 0
Trying to invert replication direction
Starting heartbeat on db1290/(none)
Verifying everything went as expected...
SUCCESS: Master switch completed successfully
Enabling GTID on old master...
Updating zarcillo...
Traceback (most recent call last):
  File "/usr/bin/db-switchover", line 33, in <module>
    sys.exit(load_entry_point('wmfmariadbpy==0.12.1', 'console_scripts', 'db-switchover')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/wmfmariadbpy/cli_admin/switchover.py", line 775, in main
    update_zarcillo(master, slave)
  File "/usr/lib/python3/dist-packages/wmfmariadbpy/cli_admin/switchover.py", line 606, in update_zarcillo
    result = zarcillo.execute(query.format(slave.host, slave.port, section, dc))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/wmfmariadbpy/WMFMariaDB.py", line 202, in execute
    cursor.execute(command)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1048, "Column 'instance' cannot be null")

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
switchover: detect if the replica is missing from zarcillorepos/sre/wmfmariadbpy!27fcerattoT430104main
db-switchover: add instance table safety checkrepos/sre/wmfmariadbpy!26fcerattoT430104-instances-checkmain
Customize query in GitLab

Event Timeline

Marostegui triaged this task as High priority.

@FCeratto-WMF please check this with high priority

FCeratto-WMF changed the task status from Open to In Progress.Jun 29 2026, 12:05 PM

I updated the PR. @Marostegui @CWilliams-WMF can we test it during a codfw switchover perhaps?

Marostegui renamed this task from db-switchover failed when insering into zarcillo to db-switchover failed when inserting into zarcillo.Jun 29 2026, 1:32 PM

@FCeratto-WMF as per the MR, my tests showed multiple issues and perhaps others still to remedy; https://phabricator.wikimedia.org/P94650

@FCeratto-WMF can we try to get this done next week? Thanks

We discussed in a meeting simplifying db-switchover but removing the SQL logic and delegating the work to the Zarcillo API; also log more clearly when the critical part of the switchover is done: SUCCESS: Master switch completed successfully