Change Details

During 2022-06-30's UTC late deployment, a config patch was synced: ``` 22:59 <+logmsgbot> !log thcipriani@deploy1002 Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:809165|Enable Wikistories on idwiki (T311143)]] (duration: 03m 31s) ``` A couple of minutes later, it was observed the change isn't fully live: ``` 23:12 <stephanebisson> thcipriani somehow my patch doesn't seem to be sync'd everywhere. When I refresh, sometimes the code is there sometimes it isn't. Is there a long replication delay or could there be a problem? ``` This affected at least mw1414 and mw1369: ``` 23:16 <urbanecm> stephanebisson: would you mind sharing which mw server works correctly and which one doesn't? should be available in the `server` header in your devtools 23:18 <stephanebisson> urbanecm mw1414.eqiad.wmnet (not updated) 23:18 <stephanebisson> urbanecm mw1413.eqiad.wmnet (updated) 23:20 <stephanebisson> urbanecm mw1369 also not up to date ``` The code on all of those servers is the same: ``` urbanecm@notebook ~ $ ssh mw1413.eqiad.wmnet md5sum /srv/mediawiki/wmf-config/InitialiseSettings.php Enter passphrase for key '/home/urbanecm/.ssh/id_ed25519_wmnet_20201001': cab73e4a083586b3f97260f3634a1414 /srv/mediawiki/wmf-config/InitialiseSettings.php urbanecm@notebook ~ $ ssh mw1414.eqiad.wmnet md5sum /srv/mediawiki/wmf-config/InitialiseSettings.php cab73e4a083586b3f97260f3634a1414 /srv/mediawiki/wmf-config/InitialiseSettings.php urbanecm@notebook ~ $ ssh mw1369.eqiad.wmnet md5sum /srv/mediawiki/wmf-config/InitialiseSettings.php cab73e4a083586b3f97260f3634a1414 /srv/mediawiki/wmf-config/InitialiseSettings.php urbanecm@notebook ~ $ ``` This is likely caused by some issue in the PHP-fpm restarts (either the restart command not arriving at all necessary hosts, or the restart script not working under certain circumstances). The same issue also happened on 2022-06-29 (see logs from https://wm-bot.wmcloud.org/browser/index.php?start=06%2F29%2F2022&end=06%2F29%2F2022&display=%23wikimedia-operations, search for `dancy` and `MatmaRex` after 14:35 on that day).

During 2022-06-30's UTC late deployment, a config patch was synced: ``` 22:59 <+logmsgbot> !log thcipriani@deploy1002 Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:809165|Enable Wikistories on idwiki (T311143)]] (duration: 03m 31s) ``` The deployment transcript is in P30698 A couple of minutes later, it was observed the change isn't fully live: ``` 23:12 <stephanebisson> thcipriani somehow my patch doesn't seem to be sync'd everywhere. When I refresh, sometimes the code is there sometimes it isn't. Is there a long replication delay or could there be a problem? ``` This affected at least mw1414 and mw1369: ``` 23:16 <urbanecm> stephanebisson: would you mind sharing which mw server works correctly and which one doesn't? should be available in the `server` header in your devtools 23:18 <stephanebisson> urbanecm mw1414.eqiad.wmnet (not updated) 23:18 <stephanebisson> urbanecm mw1413.eqiad.wmnet (updated) 23:20 <stephanebisson> urbanecm mw1369 also not up to date ``` The code on all of those servers is the same: ``` urbanecm@notebook ~ $ ssh mw1413.eqiad.wmnet md5sum /srv/mediawiki/wmf-config/InitialiseSettings.php Enter passphrase for key '/home/urbanecm/.ssh/id_ed25519_wmnet_20201001': cab73e4a083586b3f97260f3634a1414 /srv/mediawiki/wmf-config/InitialiseSettings.php urbanecm@notebook ~ $ ssh mw1414.eqiad.wmnet md5sum /srv/mediawiki/wmf-config/InitialiseSettings.php cab73e4a083586b3f97260f3634a1414 /srv/mediawiki/wmf-config/InitialiseSettings.php urbanecm@notebook ~ $ ssh mw1369.eqiad.wmnet md5sum /srv/mediawiki/wmf-config/InitialiseSettings.php cab73e4a083586b3f97260f3634a1414 /srv/mediawiki/wmf-config/InitialiseSettings.php urbanecm@notebook ~ $ ``` This is likely caused by some issue in the PHP-fpm restarts (either the restart command not arriving at all necessary hosts, or the restart script not working under certain circumstances). The same issue also happened on 2022-06-29 (see logs from https://wm-bot.wmcloud.org/browser/index.php?start=06%2F29%2F2022&end=06%2F29%2F2022&display=%23wikimedia-operations, search for `dancy` and `MatmaRex` after 14:35 on that day).