I don't know whether it's updating at deploy-time for Flow, but it's not updating normally from master: http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version .

Unrelated known problem.

In T92871#1231878, @Mattflaschen wrote:

I don't know whether it's updating at deploy-time for Flow, but it's not updating normally from master: http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version .

The Parsoid version listed on Special:Version is the extension mediawiki/extensions/Parsoid, that is a wrapper.

The beta-parsoid-update-eqiad Jenkins job pushes mediawiki/service/parsoid/ merged changes and restart the Parsoid daemon.

The bad change https://gerrit.wikimedia.org/r/#/c/194452/ was made on mediawiki/service/parsoid . I am not sure whether the beta daemon is using that repo or the deploy one. Needs to be verified.

Jenkins job beta-parsoid-update-eqiad is triggered on deployment-parsoid05.eqiad.wmflabs whenever a change on the source or deploy repository and both are rsynced respectively under:

/srv/deployment/parsoid/parsoid
/srv/deployment/parsoid/deploy

On the instance, /etc/default/parsoid has:

# Checkout of mediawiki/services/parsoid
PARSOID_BASE_PATH=/srv/deployment/parsoid/deploy/src
# Checkout of mediawiki/services/parsoid/deploy
NODE_PATH=/srv/deployment/parsoid/deploy/node_modules
# Checkout of mediawiki/services/parsoid/deploy
PARSOID_SETTINGS_FILE=/srv/deployment/parsoid/deploy/conf/wmf/betalabs.localsettings.js

That is maintained by puppet in manifests/role/parsoid.pp under the class role::parsoid::beta which has the puppet variables:

# For beta, override NODE_PATH:
$parsoid_node_path = '/srv/deployment/parsoid/deploy/node_modules'
# Also override PARSOID_SETTINGS_FILE
$parsoid_settings_file = '/srv/deployment/parsoid/deploy/conf/wmf/betalabs.localsettings.js'

# Checkout of mediawiki/services/parsoid
$parsoid_base_path = '/srv/deployment/parsoid/deploy/src'

So in short, the beta cluster is running the parsoid code from mediawiki/services/parsoid/deploy !

It used used to be source repository with the node modules from the deploy repo ie:

# Use source code from source repo
$parsoid_base_path = '/srv/deployment/parsoid/parsoid'
# And node modules from deploy repo
$parsoid_node_path = '/srv/deployment/parsoid/deploy/node_modules'

But that has been changed in October 2014 to depend solely on the deploy repo with https://gerrit.wikimedia.org/r/#/c/169622/2/manifests/role/parsoid.pp,unified

I am not sure what the Parsoid team prefers. If we want to only use the deploy repo, we need to clean up the Zuul configuration to no more deploy the mediawiki/service/parsoid repo and clean it up from the instance.

In T92871#1241795, @hashar wrote:

Jenkins job beta-parsoid-update-eqiad is triggered on deployment-parsoid05.eqiad.wmflabs whenever a change on the source or deploy repository and both are rsynced respectively under:

/srv/deployment/parsoid/parsoid
/srv/deployment/parsoid/deploy

I see ..

I am not sure what the Parsoid team prefers. If we want to only use the deploy repo, we need to clean up the Zuul configuration to no more deploy the mediawiki/service/parsoid repo and clean it up from the instance.

So, this is tricky. So, if we want beta labs to reflect code that will be deployed to production, then it has to be the deploy repo. But, if we want to test individual patches as they are merged, then we won't be able to use beta labs for any tests prior to deploy since the parsoid code will always be different from what is being deployed. @cscott thoughts?

We tend to deploy something very close to Parsoid master, so I think having the beta track master is the better option. That also gives a longer time window for people to notice beta being broken and tell us about it.

Yes, this complicates things when we deploy something which is *not* Parsoid master, perhaps because some change is risky, and it probably also complicates the case where there is a dependency between Parsoid and VE, so we need to be sure that VE is deployed to beta before the Parsoid patch lands on master. But I think we can manage that complexity.

Stated another way: the assumption is that the probability of finding a bug is proportional to the time it has live users testing it. So synching beta with master gives our buggy patches the longest possible time period to be caught before deploy.

Okay, we should also have @Catrope, @Jdforrester-WMF, @Esanders chime in as well since this also impacts VE QA on beta cluster, plus @Catrope made that original change -- so he might have had a reason for it.

@hashar so, based on IRC discussion, we want beta cluster to continue to update on parsoid patch merges. Can you do the necessary puppet changes to make that happen?

Potentially we were originally running the parsoid master branch with the node modules of the deploy repo. Maybe to ensure the tip of master is always deployable. That might explain why we had both repos cloned.

Now that we want to run the developers repo (and not the deploy ones) which node modules should we use?

1. npm clean && npm install, and thus stop triggering the job on changes made to /deploy and delete it from beta.
2. point NODE_PATH to a checkout of deploy? Might cause a bit of madness when you bump your dependencies in dev since you will need to retain back compatibility with whatever is provided by /deploy/

Both solutions are messy since npm install can install different packages than what will be deployed in production via the /deploy repo. Arlo proposed considering shrinkwrapping node modules (http://www.asyncdev.net/2012/08/locking-node-js-module-versions-with-shrinkwrap/) so that we can use solution (1) without introducing subtle failure modes because of version mismatches between beta and production. We'll discuss more tomorrow and update this here.

@ssastry any update?

Sorry .. I dropped the ball on this. Will discuss today and will update.

Let us go with solution #2 for now. Given that we don't update modules very frequently, and usually when we do, we also push updates to the deploy repo to update those modules, the times when the beta cluster is broken because of solution #2 should be infrequent. If this proves inconvenient, we can investigating shrinkwrapping.

Some of us had a meeting about it during the hackathon. The notes are on T100099 and will be discussed during June 2nd 2015 Release-Engineering-Team meeting.

IIRC the aim is to use mediawiki/services/parsoid @ master and bring the dependencies using npm.

So that is similar to T95652: On beta cluster citoid should self update and reload after change is merged where I wrote:

@mobrovac and I had some pairing last week to train him up on JJB usage. The informal .plan is to revisit each of the mediawiki services backend and have them all configured alike. Namely:

• proper source/deploy jenkins jobs
• self update via Jenkins jobs on postmerge

Change 243987 had a related patch set uploaded (by Hashar):
beta: point parsoid back to source code

https://gerrit.wikimedia.org/r/243987

@ssastry and @cscott confirmed on IRC.

I have cherry picked https://gerrit.wikimedia.org/r/#/c/243987/ on the beta cluster puppet master.

/etc/default/parsoid

- PARSOID_BASE_PATH=/srv/deployment/parsoid/deploy/src
+ PARSOID_BASE_PATH=/srv/deployment/parsoid/parsoid

Puppet restarted the service:

Scheduling refresh of Service[parsoid]

Thanks to the universal API version stuff:

curl --silent http://127.0.0.1:8000/_version|python -m json.tool
{
    "name": "parsoid",
    "sha": "9f7bdebd73d56b9059c788c94bf9cc4388c4d814",
    "version": "0.4.1-git"
}

That is the last change merged in mediawiki/parsoid.git

Change 243992 had a related patch set uploaded (by Hashar):
beta: parsoid now uses modules defined in source

https://gerrit.wikimedia.org/r/243992

Change 243997 had a related patch set uploaded (by Hashar):
parsoid: deploy node_modules with npm

https://gerrit.wikimedia.org/r/243997

I have completed the switch.

The first task earlier this week was to switch Parsoid source code from /deploy to /parsoid.git https://gerrit.wikimedia.org/r/#/c/243987/

A few minutes ago I have applied the follow up patch https://gerrit.wikimedia.org/r/#/c/243992/ which switch node_modules from /deploy to parsoid.git . That required a change to the Jenkins job https://gerrit.wikimedia.org/r/#/c/243997/ to make it run npm install.

I manually triggered the Jenkins job that update Parsoid on beta cluster (making sure it was the last merged change of parsoid.git). It managed to install dependencies:

On deployment-parsoid05 there is:

curl --silent http://127.0.0.1:8000/_version|python -m json.tool
{
    "name": "parsoid",
    "sha": "850b02d7f4d55f4d8a2496688b277c6c33a94d3b",
    "version": "0.4.1-git"
}

Change 243997 merged by jenkins-bot:
parsoid: deploy node_modules with npm

https://gerrit.wikimedia.org/r/243997

Lets keep the task open for monitoring purpose. If this need to be reverted:

• Remove https://gerrit.wikimedia.org/r/#/c/243992/ cherry pick from deployment-puppetmaster /var/lib/git/operations/puppet (or revert it in Gerrit and cherry pick the revert there).
• Revert https://gerrit.wikimedia.org/r/#/c/243997/
• Using JJB http://www.mediawiki.org/wiki/CI/JJB refresh the Jenkins job beta-parsoid-update-eqiad . You can ask assistance in #wikimedia-releng.

@subbu, @Arlolra -- at SoS release engineering said they are just waiting for us to verify that this change is effective and works well for us. We should run some tests to ensure that beta is being updated as we expect.

Current state:

$ ssh deployment-parsoid05.deployment-prep.eqiad.wmflabs 'curl --silent http://127.0.0.1:8000/_version|python -m json.tool'
{
    "name": "parsoid",
    "sha": "41f4be5b638dc6f476ca13c29bfc5cd5edff910e",
    "version": "0.4.1-git"
}

You can see the Jenkins deploy log at https://integration.wikimedia.org/ci/job/beta-parsoid-update-eqiad/ . It does not test anything and just /etc/init.d/parsoid restart. Maybe we can add to the job output the curl command above?

Soonish ™ we will migrate the Jenkins job to use scap3 but that is a different story.

Change 243987 merged by Dzahn:
beta: point parsoid back to source code

https://gerrit.wikimedia.org/r/243987

Change 243992 merged by Dzahn:
beta: parsoid now uses modules defined in source

https://gerrit.wikimedia.org/r/243992