Page MenuHomePhabricator
Create Task
Maniphest T117574

scap on beta does not sync deployment-bastion /srv/mediawiki
Closed, ResolvedPublic
Actions

Description

<Krenair> I think scap is broken on beta
<Krenair> I synchronised a file on deployment-bastion and it didn't go to the local /srv/mediawiki

jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ git diff
diff --git a/wmf-config/CommonSettings.php b/wmf-config/CommonSettings.php
index d59c6d2..87ab7fd 100755
--- a/wmf-config/CommonSettings.php
+++ b/wmf-config/CommonSettings.php
@@ -1938,7 +1938,7 @@ if ( $wmgUseVisualEditor ) {
                // HACK: $wgServerName is not available yet at this point, it's set by Setup.php
                // so use a hook
                $wgExtensionFunctions[] = function () {
-                       global $wgServerName, $wgVisualEditorRestbaseURL, $wmfRealm;
+                       global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
                        if ( $wmfRealm === 'production' ) {
                                $wgVisualEditorRestbaseURL = "https://$wgServerName/api/rest_v1/page/html/";
                                $wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ sync-file wmf-config/CommonSettings.php 
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

No syntax errors detected in /srv/mediawiki-staging/wmf-config/CommonSettings.php
20:58:09 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)                                    
20:58:19 Finished sync-masters (duration: 00m 09s)
20:58:19 Started sync-proxies
20:58:19 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/CommonSettings.php', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list.
20:58:19 Finished sync-proxies (duration: 00m 00s)
20:58:19 Started sync-apaches
sync-common: 100% (ok: 5; fail: 0; left: 0)                                     
20:58:23 Finished sync-apaches (duration: 00m 04s)
20:58:23 Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL wmf-config/*
wmf-config/CommonSettings.php:			global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL ../mediawiki/wmf-config/*
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$

Details

SubjectRepoBranchLines +/-
Add deploy masters to beta /etc/dsh/group/mediawiki-installationoperations/puppetproduction+2 -0
Customize query in gerrit

Revisions and Commits

rMSCA Scap
D33rMSCA05b2ee5d1b6a Sync to all possible dsh targets when making target list

Related Objects

Event Timeline

Krenair created this task.Nov 3 2015, 6:12 PM
Krenair raised the priority of this task from to Needs Triage.
Krenair updated the task description. (Show Details)
Krenair added a project: Beta-Cluster-Infrastructure.
Krenair added a subscriber: Krenair.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 3 2015, 6:12 PM
hashar closed this task as Resolved.Nov 3 2015, 8:56 PM
hashar claimed this task.
hashar added a subscriber: hashar.

The Jenkins job is all happy https://integration.wikimedia.org/ci/job/beta-scap-eqiad/

Krenair reopened this task as Open.Nov 3 2015, 8:57 PM

Yes, and the command I ran was happy. Doesn't mean it actually worked.

Krenair added a comment.Nov 3 2015, 8:59 PM
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ git diff
diff --git a/wmf-config/CommonSettings.php b/wmf-config/CommonSettings.php
index d59c6d2..87ab7fd 100755
--- a/wmf-config/CommonSettings.php
+++ b/wmf-config/CommonSettings.php
@@ -1938,7 +1938,7 @@ if ( $wmgUseVisualEditor ) {
                // HACK: $wgServerName is not available yet at this point, it's set by Setup.php
                // so use a hook
                $wgExtensionFunctions[] = function () {
-                       global $wgServerName, $wgVisualEditorRestbaseURL, $wmfRealm;
+                       global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
                        if ( $wmfRealm === 'production' ) {
                                $wgVisualEditorRestbaseURL = "https://$wgServerName/api/rest_v1/page/html/";
                                $wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ sync-file wmf-config/CommonSettings.php 
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

No syntax errors detected in /srv/mediawiki-staging/wmf-config/CommonSettings.php
20:58:09 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)                                    
20:58:19 Finished sync-masters (duration: 00m 09s)
20:58:19 Started sync-proxies
20:58:19 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/CommonSettings.php', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list.
20:58:19 Finished sync-proxies (duration: 00m 00s)
20:58:19 Started sync-apaches
sync-common: 100% (ok: 5; fail: 0; left: 0)                                     
20:58:23 Finished sync-apaches (duration: 00m 04s)
20:58:23 Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL wmf-config/*
wmf-config/CommonSettings.php:			global $wgServerName, $wgVisualEditorRestbaseURL, $wgVisualEditorFullRestbaseURL, $wmfRealm;
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ grep FullRestbaseURL ../mediawiki/wmf-config/*
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "https://$wgServerName/api/rest_";
../mediawiki/wmf-config/CommonSettings.php:				$wgVisualEditorFullRestbaseURL = "//$wgServerName/api/rest_";
jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$
Krenair added a project: Deployments.Nov 3 2015, 8:59 PM
hashar renamed this task from scap broken on beta? to scap on beta does not sync deployment-bastion /srv/mediawiki.Nov 3 2015, 9:13 PM
hashar removed hashar as the assignee of this task.
hashar set Security to None.
hashar updated the task description. (Show Details)
hashar added subscribers: demon, bd808.Nov 3 2015, 9:23 PM

That is a better report, I have updated the task details.

Might be caused by https://gerrit.wikimedia.org/r/#/c/224313/ Sync /srv/mediawiki-staging to co-masters (for T104826). In main.py https://gerrit.wikimedia.org/r/#/c/224313/6/scap/main.py,unified there is:

 58             # Update masters
 59             masters = self._get_master_list()
 60             with log.Timer('sync-masters', self.get_stats()):
 61                 update_masters = ssh.Job(masters, user=self.config['ssh_user'])
>62<                update_masters.exclude_hosts([socket.getfqdn()])
 63                 update_masters.command(self._master_sync_command())
 64                 update_masters.progress('sync-masters')        
 65                 succeeded, failed = update_masters.run()       
 66                 if failed:
 67                     self.get_logger().warning(
 68                         '%d masters had sync errors', failed)          
 69                     self.soft_errors = True

Line 62 has update_masters.exclude_hosts([socket.getfqdn()]) which looks like sync-masters does not sync the host on which the scap command is run.

hashar triaged this task as High priority.Nov 3 2015, 9:23 PM
hashar moved this task from To Triage to Next: Maintenance on the Beta-Cluster-Infrastructure board.
bd808 added a comment.Nov 3 2015, 9:41 PM

The local /srv/mediawiki directory on the deployment server is only ever updated by a full scap. This is how scap has always worked (at least since I ported it to python, but I think also in the old bash scripts).

bd808 added a comment.Nov 3 2015, 9:43 PM
In T117574#1779209, @hashar wrote:

Line 62 has update_masters.exclude_hosts([socket.getfqdn()]) which looks like sync-masters does not sync the host on which the scap command is run.

That code change was 100% new code and functionality so it would not have caused any regressions in existing behavior. The local master is excluded from the master-master sync step because the sync target is the same directory as the origin and thus canonically up to date.

Krenair added a comment.Nov 3 2015, 9:43 PM

Not in production:

krenair@tin:/srv/mediawiki-staging (master)$ nano README
krenair@tin:/srv/mediawiki-staging (master)$ sync-file README
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

21:42:28 Started sync-proxies
sync-proxies: 100% (ok: 12; fail: 0; left: 0)                                   
21:42:30 Finished sync-proxies (duration: 00m 01s)
21:42:30 Started sync-apaches
sync-common: 100% (ok: 467; fail: 0; left: 0)                                   
21:42:46 Finished sync-apaches (duration: 00m 16s)
21:42:46 Synchronized README: (no message) (duration: 00m 18s)
krenair@tin:/srv/mediawiki-staging (master)$ git diff
diff --git a/README b/README
index aaea9ba..723649f 100644
--- a/README
+++ b/README
@@ -1,3 +1,4 @@
+test
 operations/mediawiki-config.git
 -------------------------------
 
krenair@tin:/srv/mediawiki-staging (master)$ cat .^C
krenair@tin:/srv/mediawiki-staging (master)$ head -n 1 ../mediawiki/README
test
bd808 added a comment.Nov 3 2015, 9:48 PM

Tin is in

In T117574#1779384, @Krenair wrote:

Not in production:

tin.eqiad.wmnet is in /etc/dsh/group/mediawiki-installation in production. This is the list of hosts that are updated by the "sync-apaches" step. deployment-bastion.deployment-prep.eqiad.wmflabs is not in the file used in the beta cluster.

gerritbot added a subscriber: gerritbot.Nov 3 2015, 9:50 PM

Change 250837 had a related patch set uploaded (by BryanDavis):
Add deployment-bastion to beta /etc/dsh/group/mediawiki-installation

https://gerrit.wikimedia.org/r/250837

gerritbot added a project: Patch-For-Review.Nov 3 2015, 9:50 PM
bd808 added a comment.Nov 3 2015, 10:11 PM
jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ vim README
$ git diff README
diff --git a/README b/README
index aaea9ba..c0254ee 100644
--- a/README
+++ b/README
@@ -1,3 +1,4 @@
+test2
 operations/mediawiki-config.git
 -------------------------------

jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ sync-file README
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

22:09:28 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)
22:09:38 Finished sync-masters (duration: 00m 10s)
22:09:38 Started sync-proxies
22:09:38 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'README', 'deployment-bastion.deployment-prep.eqiad.wmflabs'] called with an empty host list.
22:09:38 Finished sync-proxies (duration: 00m 00s)
22:09:38 Started sync-apaches
sync-common: 100% (ok: 7; fail: 0; left: 0)
22:09:44 Finished sync-apaches (duration: 00m 05s)
22:09:44 Synchronized README: (no message) (duration: 00m 15s)
jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ head -n 1 ../mediawiki/README
test2

All better! :)

gerritbot added a comment.Nov 3 2015, 10:15 PM

Change 250837 merged by Rush:
Add deploy masters to beta /etc/dsh/group/mediawiki-installation

https://gerrit.wikimedia.org/r/250837

bd808 closed this task as Resolved.Nov 3 2015, 10:16 PM
bd808 claimed this task.
demon added a revision: D33: Sync to all possible dsh targets when making target list.Nov 3 2015, 10:18 PM
demon added a commit: rMSCA05b2ee5d1b6a: Sync to all possible dsh targets when making target list.Nov 5 2015, 6:39 PM
Restricted Application added a project: User-bd808. · View Herald TranscriptNov 5 2015, 6:39 PM
bd808 moved this task from To Do to Done on the User-bd808 board.Nov 6 2015, 5:07 PM
bd808 moved this task from Done to Archive on the User-bd808 board.Nov 19 2015, 4:08 PM
hashar moved this task from Next: Maintenance to Done on the Beta-Cluster-Infrastructure board.Dec 8 2015, 12:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL