Mentioned in SAL (#wikimedia-operations) [2024-10-22T14:30:06Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70506 and previous config saved to /var/cache/conftool/dbconfig/20241022-143005-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T15:30:32Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70517 and previous config saved to /var/cache/conftool/dbconfig/20241022-153031-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T16:15:53Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70524 and previous config saved to /var/cache/conftool/dbconfig/20241022-161552-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T16:16:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2172 (T367781)', diff saved to https://phabricator.wikimedia.org/P70525 and previous config saved to /var/cache/conftool/dbconfig/20241022-161604-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T16:18:16Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T367781)', diff saved to https://phabricator.wikimedia.org/P70526 and previous config saved to /var/cache/conftool/dbconfig/20241022-161816-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T17:03:38Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T367781)', diff saved to https://phabricator.wikimedia.org/P70534 and previous config saved to /var/cache/conftool/dbconfig/20241022-170337-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T17:04:01Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70535 and previous config saved to /var/cache/conftool/dbconfig/20241022-170400-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T18:04:27Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70541 and previous config saved to /var/cache/conftool/dbconfig/20241022-180426-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-22T18:49:47Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70550 and previous config saved to /var/cache/conftool/dbconfig/20241022-184946-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-24T04:58:31Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70573 and previous config saved to /var/cache/conftool/dbconfig/20241024-045830-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-24T05:58:57Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70574 and previous config saved to /var/cache/conftool/dbconfig/20241024-055856-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-24T06:44:18Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70577 and previous config saved to /var/cache/conftool/dbconfig/20241024-064418-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-24T06:45:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70578 and previous config saved to /var/cache/conftool/dbconfig/20241024-064440-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-24T07:45:07Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70579 and previous config saved to /var/cache/conftool/dbconfig/20241024-074506-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-24T08:30:36Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70582 and previous config saved to /var/cache/conftool/dbconfig/20241024-083027-arnaudb.json

@Ladsgroup for those 2 edge cases: P70585

I guess I could pre depool the nodes to let them drain ahead of time?

That log doesn't look complete, can you send the one for "T367781.log" somewhere?

Without looking at the logs, db2155 is dump replica which probably means it failed to drain it since the dumper was running (it's a known issue but hasn't been resolved) now since the dumps are paused, probably you should be able to re-run it.

Mentioned in SAL (#wikimedia-operations) [2024-10-30T07:25:21Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70675 and previous config saved to /var/cache/conftool/dbconfig/20241030-072520-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-30T08:25:47Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70682 and previous config saved to /var/cache/conftool/dbconfig/20241030-082547-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-30T09:11:09Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70689 and previous config saved to /var/cache/conftool/dbconfig/20241030-091108-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-30T09:11:32Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70690 and previous config saved to /var/cache/conftool/dbconfig/20241030-091131-arnaudb.json

In T367781#10274534, @Ladsgroup wrote:

Without looking at the logs, db2155 is dump replica which probably means it failed to drain it since the dumper was running (it's a known issue but hasn't been resolved) now since the dumps are paused, probably you should be able to re-run it.

db2155 failed to drain, db2219 went through. Will retry w/ 2155 when the other has properly repooled

Mentioned in SAL (#wikimedia-operations) [2024-10-30T09:13:44Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70691 and previous config saved to /var/cache/conftool/dbconfig/20241030-091343-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-30T09:59:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2219 (T367781)', diff saved to https://phabricator.wikimedia.org/P70694 and previous config saved to /var/cache/conftool/dbconfig/20241030-095904-arnaudb.json

In T367781#10275932, @ABran-WMF wrote:

In T367781#10274534, @Ladsgroup wrote:

Without looking at the logs, db2155 is dump replica which probably means it failed to drain it since the dumper was running (it's a known issue but hasn't been resolved) now since the dumps are paused, probably you should be able to re-run it.

db2155 failed to drain, db2219 went through. Will retry w/ 2155 when the other has properly repooled

Let me know when you start the depool, I can check what host is failing to drain so I can go and kill it.

We should try to do db2155 and db2219 as soon as we can - we should minimize the chances of getting reclones from them and "reverting" the change by cloning from the wrong host.

Mentioned in SAL (#wikimedia-operations) [2024-11-07T08:06:18Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70977 and previous config saved to /var/cache/conftool/dbconfig/20241107-080618-arnaudb.json

db2219 has been done, description updated. db2155 has been resumed, lets see if it drains properly

dangling connection is this one:

2024-11-07 08:09:19.568761 db-mysql db2155 -N -e "SELECT * FROM information_schema.processlist WHERE User like '%wiki%';"
STDOUT
2884960493      wikiadmin2023   10.194.147.226:42802    commonswiki     Sleep   509             NULL    509344.089      0       0       0.000   82352   115088  0       55992594288     NULL    0

ip shows its a wikikube pod.

cumin2024@db2155.codfw.wmnet[(none)]> show full processlist;
+------------+-----------------+----------------------+-------------+-------------+----------+---------------------------------------------------------------+-----------------------+----------+
| Id         | User            | Host                 | db          | Command     | Time     | State                                                         | Info                  | Progress |
+------------+-----------------+----------------------+-------------+-------------+----------+---------------------------------------------------------------+-----------------------+----------+
|          3 | event_scheduler | localhost            | NULL        | Daemon      | 16908477 | Waiting for next activation                                   | NULL                  |    0.000 |
|  981014295 | repl2024        | 10.192.48.208:52364  | NULL        | Binlog Dump |  7326163 | Master has sent all binlog to slave; waiting for more updates | NULL                  |    0.000 |
| 1390037031 | system user     |                      | NULL        | Slave_IO    |  4329065 | Waiting for master to send event                              | NULL                  |    0.000 |
| 1390037032 | system user     |                      | NULL        | Slave_SQL   |        0 | Slave has read all relay log; waiting for more updates        | NULL                  |    0.000 |
| 2211186136 | orchestrator    | 208.80.155.103:37206 | NULL        | Sleep       |        5 |                                                               | NULL                  |    0.000 |
| 2754985373 | orchestrator    | 208.80.155.103:35962 | NULL        | Sleep       |        5 |                                                               | NULL                  |    0.000 |
| 2754985375 | orchestrator    | 208.80.155.103:35966 | NULL        | Sleep       |        5 |                                                               | NULL                  |    0.000 |
| 2884960493 | wikiadmin2023   | 10.194.147.226:42802 | commonswiki | Sleep       |      653 |                                                               | NULL                  |    0.000 |
| 2887793366 | cumin2024       | 10.64.48.98:38182    | NULL        | Query       |        0 | starting                                                      | show full processlist |    0.000 |
+------------+-----------------+----------------------+-------------+-------------+----------+---------------------------------------------------------------+-----------------------+----------+

Nothing much outside of this

Mentioned in SAL (#wikimedia-operations) [2024-11-07T09:06:44Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70978 and previous config saved to /var/cache/conftool/dbconfig/20241107-090643-arnaudb.json

same thing happened:

Depool timed out, repooling

Mentioned in SAL (#wikimedia-operations) [2024-11-07T09:52:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P70981 and previous config saved to /var/cache/conftool/dbconfig/20241107-095205-arnaudb.json

If that is the only host pending, you can always depool it manually, run it (with no bin_log=0) and then repool it slowly.

When it can't drain, there is a maint script still attaching, here it is from 10.194.147.226. You could do a ps aux | grep -i commonswiki to see which one is like this. If it's obvious and the maint script can be killed (re-run would be fine, updateSpecialPages is one of them) just kill it.

Mentioned in SAL (#wikimedia-operations) [2024-11-12T08:03:04Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71001 and previous config saved to /var/cache/conftool/dbconfig/20241112-080303-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-12T09:03:30Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71002 and previous config saved to /var/cache/conftool/dbconfig/20241112-090329-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-12T09:48:52Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71006 and previous config saved to /var/cache/conftool/dbconfig/20241112-094851-arnaudb.json

Thanks to @CDanis for P71034 Now we know what script is refusing to update its config.

@bvibber Hiii

This script that has been running for a month and nine days. It's not done and it's not updating its config (maint scripts should call $this->waitForReplication() to update their config after each batch) and this is blocking schema changes.

• Do you have any ETA how long this script supposed to take? Maybe it's not normal to take a month to run this.
• Is it possible to add $this->waitForReplication() to the maint script so in the future, it updates its config so we can depool hosts?
• Is it okay to kill the script (i.e. is it idempotent? so it would continue from where it left off)

More details of the script:

Args:                                               
  /srv/mediawiki/multiversion/MWScript.php 
  extensions/TimedMediaHandler/maintenance/requeueTranscodes.php               
  --wiki=commonswiki       
  --throttle                                                                                                                   
  --video                                                                                                                      
  --key=144p.mjpeg.mov                                                                                                                                                                                                                                            
  --missing

How to kill it:

kubectl -n mw-script delete job mw-script.codfw.6497ohz1

thanks for the help @Ladsgroup @CDanis! @bvibber please let me know if I can help!

@ABran-WMF It really ought to have finished by now. ;) Go ahead and kill it, it's idempotent and I can clean up its state later, but I'll want to save the output log to check what went wrong.

Will do, thanks for the info @bvibber !

@bvibber: logfile is in your home directoryon deploy2002: /home/bvibber/mw-script.codfw.6497ohz1.log
job has been deleted:

kubectl --kubeconfig=admin-codfw.config  -n mw-script delete job mw-script.codfw.6497ohz1
job.batch "mw-script.codfw.6497ohz1" deleted

Will handle the last bits of this schema update

Mentioned in SAL (#wikimedia-operations) [2024-11-21T09:51:02Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71109 and previous config saved to /var/cache/conftool/dbconfig/20241121-095102-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-21T09:53:14Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71110 and previous config saved to /var/cache/conftool/dbconfig/20241121-095313-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-21T10:38:35Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71113 and previous config saved to /var/cache/conftool/dbconfig/20241121-103834-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-25T07:57:58Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2240 (T367781)', diff saved to https://phabricator.wikimedia.org/P71119 and previous config saved to /var/cache/conftool/dbconfig/20241125-075758-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-25T08:00:10Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2240 (T367781)', diff saved to https://phabricator.wikimedia.org/P71120 and previous config saved to /var/cache/conftool/dbconfig/20241125-080010-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-11-25T08:45:31Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2240 (T367781)', diff saved to https://phabricator.wikimedia.org/P71123 and previous config saved to /var/cache/conftool/dbconfig/20241125-084531-arnaudb.json