SAL
13:55 ori: [switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'".
13:56 _joe_: [switchover #3] disabling cronjobs on terbium
15:02 _joe_: [switchover #13] starting maintenace jobs
15:10 logmsgbot: ori@tin Synchronized wmf-config/ProductionServices.php: live-hack fix for rdb2*.eqiad (duration: 00m 34s)
During the switch there were times that the maint scripts were running (so the dispatcher in our case) but job queues were not able to be written to.
ie. between 13:55 and 13:56 and then between 15:02 and 15:10 (the whole windows being 13:55 to 15:10)
As far as I can tell in the case of wikidata this causes the dispatchers to run and try to queue jobs. The jobs will not get queued.
Looking at the code and speaking to @daniel this should not cause things to get missed.
However the following change was not dispatched to fiwiki
select * from wb_changes where change_revision_id = 323124588 limit 1;
+-----------+----------------------+----------------+------------------+--------------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| change_id | change_type | change_time | change_object_id | change_revision_id | change_user_id | change_info
+-----------+----------------------+----------------+------------------+--------------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 321229424 | wikibase-item~update | 20160419145156 | q1926802 | 323124588 | 1896387 | {"diff":{"type":"diff\/item","isassoc":true,"operations":{"links":{"type":"diff","isassoc":true,"operations":{"fiwiki":{"type":"diff","isassoc":true,"operations":{"name":{"type":"add","newvalue":"Michael Valgren"}}}}},"claim":{"type":"diff","isassoc":true,"operations":[]}}},"metadata":{"user_text":"Quinn","page_id":1856525,"parent_id":319636412,"comment":"\/* wbsetsitelink-add:1|fiwiki *\/ Michael Valgren","rev_id":323124588}} |
+-----------+----------------------+----------------+------------------+--------------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.03 sec)Until I manually poked the item https://www.wikidata.org/w/index.php?title=Q1926802&action=history
Another example that I have no fiddled with yet is a few rows down in the changes table.
https://www.wikidata.org/w/index.php?title=Q19933855&oldid=323124612
Link change not dispatched to https://de.wikipedia.org/wiki/Richard_Hamann_(Sanit%C3%A4tsoffizier) so no links in the side bar
| 321229448 | wikibase-item~update | 20160419145212 | q19933855 | 323124612 | 1895734 | {"diff":{"type":"diff\/item","isassoc":true,"operations":{"links":{"type":"diff","isassoc":true,"operations":{"dewiki":{"type":"diff","isassoc":true,"operations":{"name":{"type":"add","newvalue":"Richard Hamann (Sanit\u00e4tsoffizier)"}}}}},"claim":{"type":"diff","isassoc":true,"operations":[]}}},"metadata":{"user_text":"M2k~dewiki","page_id":21575717,"parent_id":316407931,"comment":"\/* wbsetsitelink-add:1|dewiki *\/ Richard Hamann (Sanit\u00e4tsoffizier)","rev_id":323124612}}Per something @daniel just said we could always take the change rows out between 2 timestamps and add them to the table again (causing them to be re dispatched)
I uses we should avoid doing this for things that have since had other changes dispatched?
I feel a maint script is needed here :)