Page MenuHomePhabricator

scap on beta does not sync deployment-bastion /srv/mediawiki
Closed, ResolvedPublic

Description

<Krenair> I think scap is broken on beta
<Krenair> I synchronised a file on deployment-bastion and it didn't go to the local /srv/mediawiki

jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ git diff
diff --git a/wmf-config/CommonSettings.php b/wmf-config/CommonSettings.php
index d59c6d2..87ab7fd 100755
--- a/wmf-config/CommonSettings.php
+++ b/wmf-config/CommonSettings.php
@@ -1938,7 +1938,7 @@ if ( $wmgUseVisualEditor ) {
                // HACK: $wgServerName is not available yet at this point, it's set by Setup.php
                // so use a hook
                $wgExtensionFunctions[] = function () {
-                       global $wgServerName, $wgVisualEditorRestbaseURL, $wmfRealm;
+                       global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
                        if ( $wmfRealm === 'production' ) {
                                $wgVisualEditorRestbaseURL = "https://$wgServerName/api/rest_v1/page/html/";
                                $wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ sync-file wmf-config/CommonSettings.php 
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

No syntax errors detected in /srv/mediawiki-staging/wmf-config/CommonSettings.php
20:58:09 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)                                    
20:58:19 Finished sync-masters (duration: 00m 09s)
20:58:19 Started sync-proxies
20:58:19 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/CommonSettings.php', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list.
20:58:19 Finished sync-proxies (duration: 00m 00s)
20:58:19 Started sync-apaches
sync-common: 100% (ok: 5; fail: 0; left: 0)                                     
20:58:23 Finished sync-apaches (duration: 00m 04s)
20:58:23 Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL wmf-config/*
wmf-config/CommonSettings.php:			global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL ../mediawiki/wmf-config/*
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$

Event Timeline

Krenair created this task.Nov 3 2015, 6:12 PM
Krenair raised the priority of this task from to Needs Triage.
Krenair updated the task description. (Show Details)
Krenair added a subscriber: Krenair.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 3 2015, 6:12 PM
hashar closed this task as Resolved.Nov 3 2015, 8:56 PM
hashar claimed this task.
hashar added a subscriber: hashar.
Krenair reopened this task as Open.Nov 3 2015, 8:57 PM

Yes, and the command I ran was happy. Doesn't mean it actually worked.

jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ git diff
diff --git a/wmf-config/CommonSettings.php b/wmf-config/CommonSettings.php
index d59c6d2..87ab7fd 100755
--- a/wmf-config/CommonSettings.php
+++ b/wmf-config/CommonSettings.php
@@ -1938,7 +1938,7 @@ if ( $wmgUseVisualEditor ) {
                // HACK: $wgServerName is not available yet at this point, it's set by Setup.php
                // so use a hook
                $wgExtensionFunctions[] = function () {
-                       global $wgServerName, $wgVisualEditorRestbaseURL, $wmfRealm;
+                       global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
                        if ( $wmfRealm === 'production' ) {
                                $wgVisualEditorRestbaseURL = "https://$wgServerName/api/rest_v1/page/html/";
                                $wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ sync-file wmf-config/CommonSettings.php 
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

No syntax errors detected in /srv/mediawiki-staging/wmf-config/CommonSettings.php
20:58:09 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)                                    
20:58:19 Finished sync-masters (duration: 00m 09s)
20:58:19 Started sync-proxies
20:58:19 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/CommonSettings.php', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list.
20:58:19 Finished sync-proxies (duration: 00m 00s)
20:58:19 Started sync-apaches
sync-common: 100% (ok: 5; fail: 0; left: 0)                                     
20:58:23 Finished sync-apaches (duration: 00m 04s)
20:58:23 Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL wmf-config/*
wmf-config/CommonSettings.php:			global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL ../mediawiki/wmf-config/*
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$
hashar renamed this task from scap broken on beta? to scap on beta does not sync deployment-bastion /srv/mediawiki.Nov 3 2015, 9:13 PM
hashar removed hashar as the assignee of this task.
hashar set Security to None.
hashar updated the task description. (Show Details)

That is a better report, I have updated the task details.

Might be caused by https://gerrit.wikimedia.org/r/#/c/224313/ Sync /srv/mediawiki-staging to co-masters (for T104826). In main.py https://gerrit.wikimedia.org/r/#/c/224313/6/scap/main.py,unified there is:

 58             # Update masters
 59             masters = self._get_master_list()
 60             with log.Timer('sync-masters', self.get_stats()):
 61                 update_masters = ssh.Job(masters, user=self.config['ssh_user'])
>62<                update_masters.exclude_hosts([socket.getfqdn()])
 63                 update_masters.command(self._master_sync_command())
 64                 update_masters.progress('sync-masters')        
 65                 succeeded, failed = update_masters.run()       
 66                 if failed:
 67                     self.get_logger().warning(
 68                         '%d masters had sync errors', failed)          
 69                     self.soft_errors = True

Line 62 has update_masters.exclude_hosts([socket.getfqdn()]) which looks like sync-masters does not sync the host on which the scap command is run.

hashar triaged this task as High priority.Nov 3 2015, 9:23 PM
hashar moved this task from To Triage to Next: Maintenance on the Beta-Cluster-Infrastructure board.

The local /srv/mediawiki directory on the deployment server is only ever updated by a full scap. This is how scap has always worked (at least since I ported it to python, but I think also in the old bash scripts).

Line 62 has update_masters.exclude_hosts([socket.getfqdn()]) which looks like sync-masters does not sync the host on which the scap command is run.

That code change was 100% new code and functionality so it would not have caused any regressions in existing behavior. The local master is excluded from the master-master sync step because the sync target is the same directory as the origin and thus canonically up to date.

Not in production:

krenair@tin:/srv/mediawiki-staging (master)$ nano README
krenair@tin:/srv/mediawiki-staging (master)$ sync-file README
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

21:42:28 Started sync-proxies
sync-proxies: 100% (ok: 12; fail: 0; left: 0)                                   
21:42:30 Finished sync-proxies (duration: 00m 01s)
21:42:30 Started sync-apaches
sync-common: 100% (ok: 467; fail: 0; left: 0)                                   
21:42:46 Finished sync-apaches (duration: 00m 16s)
21:42:46 Synchronized README: (no message) (duration: 00m 18s)
krenair@tin:/srv/mediawiki-staging (master)$ git diff
diff --git a/README b/README
index aaea9ba..723649f 100644
--- a/README
+++ b/README
@@ -1,3 +1,4 @@
+test
 operations/mediawiki-config.git
 -------------------------------
 
krenair@tin:/srv/mediawiki-staging (master)$ cat .^C
krenair@tin:/srv/mediawiki-staging (master)$ head -n 1 ../mediawiki/README
test

Tin is in

Not in production:

tin.eqiad.wmnet is in /etc/dsh/group/mediawiki-installation in production. This is the list of hosts that are updated by the "sync-apaches" step. deployment-bastion.deployment-prep.eqiad.wmflabs is not in the file used in the beta cluster.

Change 250837 had a related patch set uploaded (by BryanDavis):
Add deployment-bastion to beta /etc/dsh/group/mediawiki-installation

https://gerrit.wikimedia.org/r/250837

jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ vim README
$ git diff README
diff --git a/README b/README
index aaea9ba..c0254ee 100644
--- a/README
+++ b/README
@@ -1,3 +1,4 @@
+test2
 operations/mediawiki-config.git
 -------------------------------

jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ sync-file README
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

22:09:28 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)
22:09:38 Finished sync-masters (duration: 00m 10s)
22:09:38 Started sync-proxies
22:09:38 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'README', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list.
22:09:38 Finished sync-proxies (duration: 00m 00s)
22:09:38 Started sync-apaches
sync-common: 100% (ok: 7; fail: 0; left: 0)
22:09:44 Finished sync-apaches (duration: 00m 05s)
22:09:44 Synchronized README: (no message) (duration: 00m 15s)
jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ head -n 1 ../mediawiki/README
test2

All better! :)

Change 250837 merged by Rush:
Add deploy masters to beta /etc/dsh/group/mediawiki-installation

https://gerrit.wikimedia.org/r/250837

bd808 closed this task as Resolved.Nov 3 2015, 10:16 PM
bd808 claimed this task.
Restricted Application added a project: User-bd808. · View Herald TranscriptNov 5 2015, 6:39 PM
bd808 moved this task from To Do to Done on the User-bd808 board.Nov 6 2015, 5:07 PM
bd808 moved this task from Done to Archive on the User-bd808 board.Nov 19 2015, 4:08 PM