Page MenuHomePhabricator

Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code
Closed, ResolvedPublic


We had a bad patch (with a typo) get merged in Parsoid (, but Beta Cluster wasn't updated to this patch. It was only after a deploy repo patch was merged this morning ( that beta labs got updated and Flow noticed that Parsoid instance broke there and filed T92859.

Without that breakage, we would have ended up deploying this code to production this afternoon and would have had to revert after breaking production Parsoid.

We are also missing mock statsd based testing of this code in Parsoid itself -- that will be fixed separately.

Event Timeline

ssastry raised the priority of this task from to High.
ssastry updated the task description. (Show Details)
ssastry added a subscriber: ssastry.
greg renamed this task from Parsoid patches don't update beta labs automatically -- only deploy repo patches seem to update that code to Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code.Mar 16 2015, 6:24 PM
greg updated the task description. (Show Details)
greg set Security to None.

I don't know whether it's updating at deploy-time for Flow, but it's not updating normally from master: .

In T92871#1231878, @Mattflaschen wrote:

I don't know whether it's updating at deploy-time for Flow, but it's not updating normally from master: .

The Parsoid version listed on Special:Version is the extension mediawiki/extensions/Parsoid, that is a wrapper.

The beta-parsoid-update-eqiad Jenkins job pushes mediawiki/service/parsoid/ merged changes and restart the Parsoid daemon.

The bad change was made on mediawiki/service/parsoid . I am not sure whether the beta daemon is using that repo or the deploy one. Needs to be verified.

Jenkins job beta-parsoid-update-eqiad is triggered on deployment-parsoid05.eqiad.wmflabs whenever a change on the source or deploy repository and both are rsynced respectively under:


On the instance, /etc/default/parsoid has:

# Checkout of mediawiki/services/parsoid
# Checkout of mediawiki/services/parsoid/deploy
# Checkout of mediawiki/services/parsoid/deploy

That is maintained by puppet in manifests/role/parsoid.pp under the class role::parsoid::beta which has the puppet variables:

# For beta, override NODE_PATH:
$parsoid_node_path = '/srv/deployment/parsoid/deploy/node_modules'
$parsoid_settings_file = '/srv/deployment/parsoid/deploy/conf/wmf/betalabs.localsettings.js'

# Checkout of mediawiki/services/parsoid
$parsoid_base_path = '/srv/deployment/parsoid/deploy/src'

So in short, the beta cluster is running the parsoid code from mediawiki/services/parsoid/deploy !

It used used to be source repository with the node modules from the deploy repo ie:

# Use source code from source repo
$parsoid_base_path = '/srv/deployment/parsoid/parsoid'
# And node modules from deploy repo
$parsoid_node_path = '/srv/deployment/parsoid/deploy/node_modules'

But that has been changed in October 2014 to depend solely on the deploy repo with,unified

I am not sure what the Parsoid team prefers. If we want to only use the deploy repo, we need to clean up the Zuul configuration to no more deploy the mediawiki/service/parsoid repo and clean it up from the instance.

hashar moved this task from Non-Parsing-Team Tasks to Needs Discussion on the Parsoid board.
hashar moved this task from Backlog to In progress on the Continuous-Integration-Config board.

Jenkins job beta-parsoid-update-eqiad is triggered on deployment-parsoid05.eqiad.wmflabs whenever a change on the source or deploy repository and both are rsynced respectively under:


I see ..

I am not sure what the Parsoid team prefers. If we want to only use the deploy repo, we need to clean up the Zuul configuration to no more deploy the mediawiki/service/parsoid repo and clean it up from the instance.

So, this is tricky. So, if we want beta labs to reflect code that will be deployed to production, then it has to be the deploy repo. But, if we want to test individual patches as they are merged, then we won't be able to use beta labs for any tests prior to deploy since the parsoid code will always be different from what is being deployed. @cscott thoughts?

We tend to deploy something very close to Parsoid master, so I think having the beta track master is the better option. That also gives a longer time window for people to notice beta being broken and tell us about it.

Yes, this complicates things when we deploy something which is *not* Parsoid master, perhaps because some change is risky, and it probably also complicates the case where there is a dependency between Parsoid and VE, so we need to be sure that VE is deployed to beta before the Parsoid patch lands on master. But I think we can manage that complexity.

Stated another way: the assumption is that the probability of finding a bug is proportional to the time it has live users testing it. So synching beta with master gives our buggy patches the longest possible time period to be caught before deploy.

Okay, we should also have @Catrope, @Jdforrester-WMF, @Esanders chime in as well since this also impacts VE QA on beta cluster, plus @Catrope made that original change -- so he might have had a reason for it.

@hashar so, based on IRC discussion, we want beta cluster to continue to update on parsoid patch merges. Can you do the necessary puppet changes to make that happen?

Potentially we were originally running the parsoid master branch with the node modules of the deploy repo. Maybe to ensure the tip of master is always deployable. That might explain why we had both repos cloned.

Now that we want to run the developers repo (and not the deploy ones) which node modules should we use?

  1. npm clean && npm install, and thus stop triggering the job on changes made to /deploy and delete it from beta.
  2. point NODE_PATH to a checkout of deploy? Might cause a bit of madness when you bump your dependencies in dev since you will need to retain back compatibility with whatever is provided by /deploy/

Both solutions are messy since npm install can install different packages than what will be deployed in production via the /deploy repo. Arlo proposed considering shrinkwrapping node modules ( so that we can use solution (1) without introducing subtle failure modes because of version mismatches between beta and production. We'll discuss more tomorrow and update this here.

Sorry .. I dropped the ball on this. Will discuss today and will update.

Let us go with solution #2 for now. Given that we don't update modules very frequently, and usually when we do, we also push updates to the deploy repo to update those modules, the times when the beta cluster is broken because of solution #2 should be infrequent. If this proves inconvenient, we can investigating shrinkwrapping.

Some of us had a meeting about it during the hackathon. The notes are on T100099 and will be discussed during June 2nd 2015 Release-Engineering-Team meeting.

IIRC the aim is to use mediawiki/services/parsoid @ master and bring the dependencies using npm.

So that is similar to T95652: On beta cluster citoid should self update and reload after change is merged where I wrote:

@mobrovac and I had some pairing last week to train him up on JJB usage. The informal .plan is to revisit each of the mediawiki services backend and have them all configured alike. Namely:

  • proper source/deploy jenkins jobs
  • self update via Jenkins jobs on postmerge
hashar lowered the priority of this task from High to Medium.Jul 6 2015, 7:37 PM
hashar moved this task from To Triage to In-progress on the Beta-Cluster-Infrastructure board.

Was bring up during the SoS. An idea was to overhaul all the services deployment ( T100099 ) but we eventually lost track of the idea.

Change 243987 had a related patch set uploaded (by Hashar):
beta: point parsoid back to source code

@ssastry and @cscott confirmed on IRC.

I have cherry picked on the beta cluster puppet master.


- PARSOID_BASE_PATH=/srv/deployment/parsoid/deploy/src
+ PARSOID_BASE_PATH=/srv/deployment/parsoid/parsoid

Puppet restarted the service:

Scheduling refresh of Service[parsoid]

Thanks to the universal API version stuff:

curl --silent|python -m json.tool
    "name": "parsoid",
    "sha": "9f7bdebd73d56b9059c788c94bf9cc4388c4d814",
    "version": "0.4.1-git"

That is the last change merged in mediawiki/parsoid.git

Change 243992 had a related patch set uploaded (by Hashar):
beta: parsoid now uses modules defined in source

Change 243997 had a related patch set uploaded (by Hashar):
parsoid: deploy node_modules with npm

hashar raised the priority of this task from Medium to High.Oct 6 2015, 8:06 PM

Current status:

  • Parsoid on beta cluster runs code from mediawiki/services/parsoid with dependencies from mediawiki/services/deploy

Pending reviews / double check etc.

I have completed the switch.

The first task earlier this week was to switch Parsoid source code from /deploy to /parsoid.git

A few minutes ago I have applied the follow up patch which switch node_modules from /deploy to parsoid.git . That required a change to the Jenkins job to make it run npm install.

I manually triggered the Jenkins job that update Parsoid on beta cluster (making sure it was the last merged change of parsoid.git). It managed to install dependencies:

1Started by user hashar
2Building remotely on deployment-parsoid05 (deployment-parsoid-eqiad) in workspace /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad
3[beta-parsoid-update-eqiad] $ /bin/bash -xe /tmp/
4+ /srv/deployment/integration/slave-scripts/bin/ mediawiki/services/parsoid mediawiki/services/parsoid/deploy
5Change triggered by
7Triggering informations:
8Pipeline : postmerge
9Project : mediawiki/services/parsoid
10Branch : master
12Hostname : deployment-parsoid05.deployment-prep.eqiad.wmflabs
14Refreshing mediawiki/services/parsoid in /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/parsoid
15Now in directory /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/parsoid
16Applying patch refs/zuul/master/Z1d2dbcf89a6a469b9f9f01e2482f3cd8 to mediawiki/services/parsoid in /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/parsoid
17From git://zuul.eqiad.wmnet/mediawiki/services/parsoid
18 * branch refs/zuul/master/Z1d2dbcf89a6a469b9f9f01e2482f3cd8 -> FETCH_HEAD
19+ git checkout -f FETCH_HEAD
20HEAD is now at 850b02d... An inline_break is a fine way to end a list
21+ git reset --hard FETCH_HEAD
22HEAD is now at 850b02d An inline_break is a fine way to end a list
23+ git clean -xdff -q
24+ set +x
25Refreshing mediawiki/services/parsoid submodules...
26Done refreshing and patching mediawiki/services/parsoid
28Refreshing mediawiki/services/parsoid/deploy in /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/deploy
29Now in directory /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/deploy
30Attempting to get mediawiki/services/parsoid/deploy to match change branch 'master'
31Switching mediawiki/services/parsoid/deploy to master
32+ git checkout -f master
33Already on 'master'
34Your branch is up-to-date with 'origin/master'.
35+ git reset --hard remotes/origin/master
36HEAD is now at 3129cbb Bump src to 62971510b for deploy
37+ git clean -xdff -q
38+ set +x
39Refreshing mediawiki/services/parsoid/deploy submodules...
40Done refreshing mediawiki/services/parsoid/deploy
42[beta-parsoid-update-eqiad] $ /bin/bash -xe /tmp/
43+ cd parsoid
44+ rm -fR node_modules
45+ npm install
46npm http GET
47npm http GET
48npm http GET
49npm http GET
50npm http GET
51npm http GET
52npm http GET
53npm http GET
54npm http GET
55npm http GET
56npm http GET
57npm http GET
58npm http GET
59npm http GET
60npm http GET
61npm http GET
62npm http GET
63npm http GET
64npm http GET
65npm http GET
66npm http GET
67npm http GET
68npm http GET
69npm http GET
70npm http GET
71npm http GET
72npm http GET
73npm http GET
74npm http GET
75npm http 200
76npm http 200
77npm http 200
78npm http 200
79npm http 200
80npm http 200
81npm http 200
82npm http 200
83npm http 200
84npm http 200
85npm http 200
86npm http 200
87npm http 200
88npm http 200
89npm http 200
90npm http 200
91npm http 200
92npm http 200
93npm http 200
94npm http 200
95npm http 200
96npm http 200
97npm http 200
98npm http 200
99npm http 200
100npm http 200
101npm http 200
102npm http 200
103npm http 200
104npm http GET
105npm http GET
106npm http GET
107npm http GET
108npm http GET
109npm http GET
110npm http GET
111npm http GET
112npm http GET
113npm http GET
114npm http GET
115npm http GET
116npm http 200
117npm http 200
118npm http 200
119npm http 200
120npm http 200
121npm http 200
122npm http 200
123npm http 200
124npm http 200
125npm http 200
126npm http 200
127npm http 200
128npm http GET
129npm http GET
130npm http GET
131npm http GET
132npm http GET
133npm http GET
134npm http GET
135npm http GET
136npm http GET
137npm http GET
138npm http GET
139npm http GET
140npm http GET
141npm http 200
142npm http 200
143npm http 200
144npm http 200
145npm http 200
146npm http 200
147npm http 200
148npm http 200
149npm http 200
150npm http 200
151npm http 200
152npm http 200
153npm http 200
154npm http GET
155npm http GET
156npm http 200
157npm http 200
158npm http GET
159npm http GET
160npm http GET
161npm http GET
162npm http 200
163npm http 200
164npm http 200
165npm http 200
166npm http GET
167npm http GET
168npm http GET
169npm http GET
170npm http GET
171npm http GET
172npm http GET
173npm http GET
174npm http GET
175npm http GET
176npm http GET
177npm http GET
178npm http GET
179npm http GET
180npm http GET
181npm http GET
182npm http GET
183npm http GET
184npm http GET
185npm http GET
186npm http GET
187npm http GET
188npm http 200
189npm http 200
190npm http 200
191npm http 200
192npm http 200
193npm http 200
194npm http 200
195npm http 200
196npm http 200
197npm http 200
198npm http 200
199npm http 200
200npm http 200
201npm http 200
202npm http 200
203npm http 200
204npm http 200
205npm http 200
206npm http 200
207npm http 200
208npm http 200
209npm http 200
210npm http GET
211npm http GET
212npm http GET
213npm http GET
214npm http GET
215npm http GET
216npm http GET
217npm http GET
218npm http GET
219npm http GET
220npm http GET
221npm http GET
222npm http GET
223npm http GET
224npm http GET
225npm http GET
226npm http GET
227npm http GET
228npm http 200
229npm http 200
230npm http 200
231npm http 200
232npm http 200
233npm http 200
234npm http 200
235npm http 200
236npm http 200
237npm http 200
238npm http 200
239npm http 200
240npm http 200
241npm http 200
242npm http 200
243npm http 200
244npm http 200
245npm http 200
246npm http GET
247npm http 200
248npm http GET
249npm http GET
250npm http 200
251npm http 200
252npm http GET
253npm http GET
254npm http 200
255npm http 200
256npm http GET
257npm http GET
258npm http GET
259npm http GET
260npm http GET
261npm http GET
262npm http GET
263npm http GET
264npm http GET
265npm http GET
266npm http GET
267npm http GET
268npm http GET
269npm http GET
270npm http GET
271npm http GET
272npm http GET
273npm http GET
274npm http GET
275npm http GET
276npm http GET
277npm http GET
278npm http GET
279npm http 200
280npm http 200
281npm http 200
282npm http 200
283npm http 200
284npm http 200
285npm http 200
286npm http 200
287npm http 200
288npm http 200
289npm http 200
290npm http 200
291npm http 200
292npm http 200
293npm http 200
294npm http 200
295npm http 200
296npm http 200
297npm http 200
298npm http 200
299npm http 200
300npm http 200
301npm http 200
302npm http GET
303npm http GET
304npm http GET
305npm http GET
306npm http GET
307npm http 200
308npm http 200
309npm http 200
310npm http 200
311npm http 200
312npm http GET
313npm http GET
314npm http GET
315npm http GET
316npm http GET
317npm http GET
318npm http GET
319npm http GET
320npm http GET
321npm http 200
322npm http 200
323npm http 200
324npm http 200
325npm http 200
326npm http 200
327npm http 200
328npm http 200
329npm http 200
330npm http GET
331npm http 200
332npm http GET
333npm http 200
334npm http GET
335npm http GET
336npm http GET
337npm http GET
338npm http GET
339npm http GET
340npm http GET
341npm http GET
342npm http GET
343npm http GET
344npm http 200
345npm http 200
346npm http 200
347npm http 200
348npm http 200
349npm http 200
350npm http 200
351npm http 200
352npm http 200
353npm http 200
354npm http GET
355npm http 200
356npm http GET
357npm http GET
358npm http 200
359npm http 200
360npm http GET
361npm http GET
362npm http GET
363npm http GET
364npm http GET
365npm http GET
366npm http GET
367npm http GET
368npm http GET
369npm http GET
370npm http GET
371npm http 200
372npm http 200
373npm http 200
374npm http 200
375npm http 200
376npm http 200
377npm http 200
378npm http 200
379npm http 200
380npm http 200
381npm http 200
382npm http GET
383npm http GET
384npm http GET
385npm http 200
386npm http 200
387npm http 200
388npm http GET
389npm http GET
390npm http GET
391npm http GET
392npm http GET
393npm http GET
394npm http 200
395npm http 200
396npm http 200
397npm http 200
398npm http 200
399npm http 200
400npm http GET
401npm http 200
402npm http GET
403npm http 200
404npm http GET
405npm http GET
406npm http GET
407npm http 200
408npm http 200
409npm http 200
410npm http GET
411npm http 200
412npm http GET
413npm http 200
414npm http GET
415npm http GET
416npm http GET
417npm http GET
418npm http 200
419npm http 200
420npm http 200
421npm http GET
422npm http GET
423npm http 200
424npm http 200
425npm http 200
426npm http GET
427npm http GET
428npm http 200
429npm http 200
430npm http GET
431npm http GET
432npm http 200
433npm http 200
434npm WARN engine hawk@3.1.0: wanted: {"node":">=0.10.32"} (current: {"node":"v0.10.25","npm":"1.3.10"})
435npm http GET
436npm http GET
437npm http GET
438npm http 200
439npm http 200
440npm http GET
441npm http GET
442npm http 200
443npm http 200
444npm http 200
445npm http GET
446npm http GET
447npm http GET
448npm http GET
449npm http GET
450npm http GET
451npm http GET
452npm http GET
453npm http 200
454npm http 200
455npm http 200
456npm http 200
457npm http 200
458npm http 200
459npm http 200
460npm http 200
461npm http GET
462npm http GET
463npm http 200
464npm http 200
465npm http GET
466npm http 200
467npm http GET
468npm http GET
469npm http GET
470npm http GET
471npm http GET
472npm http GET
473npm http GET
474npm http GET
475npm http GET
476npm http GET
477npm http GET
478npm http GET
479npm http 200
480npm http 200
481npm http 200
482npm http 200
483npm http 200
484npm http 200
485npm http 200
486npm http 200
487npm http 200
488npm http 200
489npm http 200
490npm http 200
491npm http GET
492npm http GET
493npm http GET
494npm http GET
495npm http 200
496npm http 200
497npm http GET
498npm http 200
499npm http 200
500npm http 200
501npm http GET
502npm http GET
503npm http 200
504npm http 200
505npm http GET
506npm http GET
507npm http GET
508npm http GET
509npm http GET
510npm http GET
511npm http 200
512npm http 200
513npm http 200
514npm http 200
515npm http 200
516npm http 200
517npm WARN engine cryptiles@2.0.5: wanted: {"node":">=0.10.40"} (current: {"node":"v0.10.25","npm":"1.3.10"})
518npm WARN engine boom@2.9.0: wanted: {"node":">=0.10.40"} (current: {"node":"v0.10.25","npm":"1.3.10"})
519npm WARN engine hoek@2.16.3: wanted: {"node":">=0.10.40"} (current: {"node":"v0.10.25","npm":"1.3.10"})
520npm http GET
521npm http GET
522npm http 200
523npm http GET
524npm http GET
525npm http GET
526npm http GET
527npm http GET
528npm http GET
529npm http GET
530npm http 200
531npm http 200
532npm http 200
533npm http 200
534npm http 200
535npm http 200
536npm http 200
537npm http 200
538npm http GET
539npm http 200
540npm http GET
541npm http GET
542npm http 200
543npm http 200
544npm http GET
545npm http GET
546npm http GET
547npm http GET
548npm http GET
549npm http 200
550npm http 200
551npm http 200
552npm http 200
553npm http 200
554npm http GET
555npm http GET
556npm http GET
557npm http GET
558npm http GET
559npm http GET
560npm http GET
561npm http GET
562npm http GET
563npm http GET
564npm http 200
565npm http 200
566npm http 200
567npm http 200
568npm http 200
569npm http 200
570npm http 200
571npm http 200
572npm http 200
573npm http 200
574npm http GET
575npm http 200
576npm http GET
577npm http GET
578npm http 200
579npm http GET
580npm http 200
581npm http 200
582npm http GET
583npm http 200
584npm http GET
585npm http GET
586npm http 200
587npm http 200
588npm http GET
589npm http GET
590npm http GET
591npm http GET
592npm http GET
593npm http GET
594npm http 200
595npm http 200
596npm http 200
597npm http 200
598npm http 200
599npm http 200
600npm http GET
601npm http GET
602npm http GET
603npm http 200
604npm http 200
605npm http 200
606npm http GET
607npm http GET
608npm http 200
609npm http 200
610npm http GET
611npm http GET
612npm http 200
613npm http 200
614npm http GET
615npm http GET
616npm http GET
617npm http 200
618npm http 200
619npm http 200
620npm http GET
621npm http GET
622npm http GET
623npm http GET
624npm http GET
625npm http GET
626npm http GET
627npm http 200
628npm http 200
629npm http GET
630npm http GET
631npm http GET
632npm http GET
633npm http GET
634npm http GET
635npm http GET
636npm http GET
637npm http GET
638npm http GET
639npm http GET
640npm http GET
641npm http GET
642npm http GET
643npm http GET
644npm http GET
645npm http GET
646npm http 200
647npm http 200
648npm http 200
649npm http 200
650npm http 200
651npm http 200
652npm http 200
653npm http 200
654npm http 200
655npm http 200
656npm http 200
657npm http 200
658npm http 200
659npm http 200
660npm http 200
661npm http 200
662npm http 200
663npm http 200
664npm http 200
665npm http 200
666npm http 200
667npm http 200
668npm http GET
669npm http GET
670npm http 200
671npm http 200
672npm http GET
673npm http 200
674npm http GET
675npm http GET
676npm http 200
677npm http 200
678npm http GET
679npm http 200
680npm http GET
681npm http GET
682npm http GET
683npm http GET
684npm http GET
685npm http 200
686npm http 200
687npm http 200
688npm http 200
689npm http 200
690npm http GET
691npm http 200
692npm http GET
693npm http GET
694npm http 200
695npm http 200
696npm http GET
697npm http GET
698npm http GET
699npm http 200
700npm http 200
701npm http 200
702npm http GET
703npm http GET
704npm http GET
705npm http GET
706npm http GET
707npm http 200
708npm http 200
709npm http 200
710npm http 200
711npm http 200
712npm http GET
713npm http GET
714npm http 200
715npm http 200
716npm http GET
717npm http GET
718npm http 200
719npm http 200
720npm http GET
721npm http GET
722npm http GET
723npm http GET
724npm http 200
725npm http 200
726npm http 200
727npm http 200
728npm http GET
729npm http GET
730npm http 200
731npm http 200
732npm http GET
733npm http GET
734npm http 200
735npm http 200
736npm http GET
737npm http GET
738npm http GET
739npm http GET
740npm http GET
741npm http GET
742npm http GET
743npm http GET
744npm http GET
745npm http GET
746npm http GET
747npm http GET
748npm http GET
749npm http GET
750npm http GET
751npm http 200
752npm http 200
753npm http 200
754npm http 200
755npm http 200
756npm http 200
757npm http 200
758npm http 200
759npm http 200
760npm http 200
761npm http 200
762npm http 200
763npm http 200
764npm http 200
765npm http 200
766npm http GET
767npm http GET
768npm http GET
769npm http GET
770npm http GET
771npm http 200
772npm http 200
773npm http 200
774npm http 200
775npm http 200
776npm http GET
777npm http 200
778npm http GET
779npm http GET
780npm http GET
781npm http GET
782npm http GET
783npm http 200
784npm http GET
785npm http GET
786npm http 200
787npm http 200
788npm http 200
789npm http 200
790npm http GET
791npm http GET
792npm http GET
793npm http GET
794npm http GET
795npm http GET
796npm http GET
797npm http 200
798npm http 200
799npm http 200
800npm http 200
801npm http 200
802npm http GET
803npm http GET
804npm http GET
805npm http GET
806npm http GET
807npm http 200
808npm http 200
809npm http 200
810npm http 200
811npm http 200
812npm http 200
813npm http 200
814npm http 200
815npm http 200
816npm http GET
817npm http 200
818npm http GET
819npm http GET
820npm http GET
821npm http GET
822npm http GET
823npm http 200
824npm http 200
825npm http 200
826npm http GET
827npm http GET
828npm http GET
829npm http GET
830npm http 200
831npm http 200
832npm http 200
833npm http 200
834npm http 200
835npm http 200
836npm http GET
837npm http GET
838npm http 200
839npm http 200
840npm http GET
841npm http 200
842npm http GET
843npm http GET
844npm http GET
845npm http GET
846npm http GET
847npm http GET
848npm http 200
849npm http GET
850npm http 200
851npm http 200
852npm http 200
853npm http 200
854npm http 200
855npm http 200
856npm http GET
857npm http GET
858npm http GET
859npm http GET
860npm http GET
861npm http 200
862npm http 200
863npm http 200
864npm http 200
865npm http 200
866npm http GET
867npm http 200
868npm http GET
869npm http GET
870npm http GET
871npm http GET
872npm http GET
873npm http GET
874npm http GET
875npm http GET
876npm http GET
877npm http GET
878npm http GET
879npm http GET
880npm http GET
881npm http GET
882npm http GET
883npm http GET
884npm http GET
885npm http GET
886npm http GET
887npm http GET
888npm http GET
889npm http GET
890npm http GET
891npm http GET
892npm http GET
893npm http GET
894npm http GET
895npm http GET
896npm http GET
897npm http GET
898npm http GET
899npm http GET
900npm http GET
901npm http GET
902npm http GET
903npm http GET
904npm http GET
905npm http GET
906npm http GET
907npm http GET
908npm http GET
909npm http GET
910npm http GET
911npm http GET
912npm http 200
913npm http 200
914npm http 200
915npm http 200
916npm http 200
917npm http 200
918npm http 200
919npm http 200
920npm http 200
921npm http 200
922npm http 200
923npm http 200
924npm http 200
925npm http 200
926npm http 200
927npm http 200
928npm http 200
929npm http 200
930npm http 200
931npm http 200
932npm http 200
933npm http 200
934npm http 200
935npm http 200
936npm http 200
937npm http 200
938npm http 200
939npm http 200
940npm http 200
941npm http 200
942npm http 200
943npm http 200
944npm http 200
945npm http 200
946npm http 200
947npm http 200
948npm http 200
949npm http 200
950npm http 200
951npm http 200
952npm http 200
953npm http 200
954npm http 200
955npm http 200
956npm http GET
957npm http GET
958npm http GET
959npm http GET
960npm http 200
961npm http 200
962npm http GET
963npm http GET
964npm http 200
965npm http GET
966npm http GET
967npm http 200
968npm http 200
969npm http GET
970npm http 200
971npm http 200
972npm http 200
973npm http 200
974npm http GET
975npm http 200
976npm http GET
977npm http 200
978npm http GET
979npm http GET
980npm http GET
981npm http GET
982npm http 200
983npm http 200
984npm http 200
985npm http 200
986npm http GET
987npm http 200
988npm http GET
989npm http GET
990npm http GET
991npm http GET
992npm http GET
993npm http 200
994npm http 200
995npm http 200
996npm http 200
997npm http 200
998npm http GET
999npm http 200
1000npm http GET
1001npm http 200
1002npm http GET
1003npm http 200
1004npm http GET
1005npm http 200
1006npm http GET
1007npm http 200
1008npm http GET
1009npm http GET
1010npm http 200
1011npm http 200
1012npm http GET
1013npm http 200
1014npm http GET
1015npm http GET
1016npm http 200
1017npm http 200
1018npm http GET
1019npm http 200
1020npm http GET
1021npm http 200
1022npm http GET
1023npm http GET
1024npm http GET
1025npm http GET
1026npm http GET
1027npm http GET
1028npm http GET
1029npm http 200
1030npm http 200
1031npm http 200
1032npm http 200
1033npm http 200
1034npm http 200
1035npm http 200
1036npm http GET
1037npm http GET
1038npm http GET
1039npm http GET
1040npm http GET
1041npm http GET
1042npm http GET
1043npm http GET
1044npm http GET
1045npm http GET
1046npm http 200
1047npm http 200
1048npm http 200
1049npm http 200
1050npm http 200
1051npm http 200
1052npm http 200
1053npm http 200
1054npm http 200
1055npm http 200
1056npm http GET
1057npm http GET
1058npm http GET
1059npm http GET
1060npm http 200
1061npm http 200
1062npm http 200
1063npm http 200
1064npm http GET
1065npm http 200
1066npm http GET
1067npm http 200
1068npm http GET
1069npm http GET
1070npm http 200
1071npm http 200
1072npm http GET
1073npm http 200
1074simplediff@0.1.1 node_modules/simplediff
1076diff@1.0.7 node_modules/diff
1078node-txstatsd@0.1.5 node_modules/node-txstatsd
1080alea@0.0.9 node_modules/alea
1082node-uuid@1.4.1 node_modules/node-uuid
1084colors@1.1.2 node_modules/colors
1086entities@1.1.1 node_modules/entities
1088async@0.9.0 node_modules/async
1090gelf-stream@0.2.4 node_modules/gelf-stream
1091└── gelfling@0.2.0
1093serve-favicon@2.3.0 node_modules/serve-favicon
1094├── ms@0.7.1
1095├── fresh@0.3.0
1096├── etag@1.7.0
1097└── parseurl@1.3.0
1099yargs@1.3.1 node_modules/yargs
1101pegjs@0.8.0 node_modules/pegjs
1103chai@3.0.0 node_modules/chai
1104├── assertion-error@1.0.1
1105├── type-detect@1.0.0
1106└── deep-eql@0.1.3 (type-detect@0.1.1)
1108compression@1.5.2 node_modules/compression
1109├── bytes@2.1.0
1110├── vary@1.0.1
1111├── on-headers@1.0.0
1112├── debug@2.2.0 (ms@0.7.1)
1113├── compressible@2.0.5 (mime-db@1.17.0)
1114└── accepts@1.2.12 (negotiator@0.5.3, mime-types@2.1.5)
1116express@4.13.3 node_modules/express
1117├── escape-html@1.0.2
1118├── merge-descriptors@1.0.0
1119├── array-flatten@1.1.1
1120├── cookie@0.1.3
1121├── path-to-regexp@0.1.7
1122├── utils-merge@1.0.0
1123├── cookie-signature@1.0.6
1124├── methods@1.1.1
1125├── fresh@0.3.0
1126├── range-parser@1.0.2
1127├── vary@1.0.1
1128├── content-type@1.0.1
1129├── etag@1.7.0
1130├── parseurl@1.3.0
1131├── content-disposition@0.5.0
1132├── serve-static@1.10.0
1133├── depd@1.0.1
1134├── qs@4.0.0
1135├── on-finished@2.3.0 (ee-first@1.1.1)
1136├── finalhandler@0.4.0 (unpipe@1.0.0)
1137├── debug@2.2.0 (ms@0.7.1)
1138├── proxy-addr@1.0.8 (forwarded@0.1.0, ipaddr.js@1.0.1)
1139├── send@0.13.0 (destroy@1.0.3, ms@0.7.1, statuses@1.2.1, mime@1.3.4, http-errors@1.3.1)
1140├── accepts@1.2.12 (negotiator@0.5.3, mime-types@2.1.5)
1141└── type-is@1.6.7 (media-typer@0.3.0, mime-types@2.1.5)
1143bunyan@1.0.0 node_modules/bunyan
1144└── mv@2.0.3 (rimraf@2.2.8, ncp@0.6.0, mkdirp@0.5.0)
1146prfun@2.0.0 node_modules/prfun
1147└── es6-shim@0.33.3
1149body-parser@1.13.3 node_modules/body-parser
1150├── bytes@2.1.0
1151├── content-type@1.0.1
1152├── depd@1.0.1
1153├── qs@4.0.0
1154├── on-finished@2.3.0 (ee-first@1.1.1)
1155├── raw-body@2.1.2 (unpipe@1.0.0)
1156├── http-errors@1.3.1 (inherits@2.0.1, statuses@1.2.1)
1157├── debug@2.2.0 (ms@0.7.1)
1158├── iconv-lite@0.4.11
1159└── type-is@1.6.7 (media-typer@0.3.0, mime-types@2.1.5)
1161mocha@2.2.5 node_modules/mocha
1162├── escape-string-regexp@1.0.2
1163├── supports-color@1.2.1
1164├── growl@1.8.1
1165├── commander@2.3.0
1166├── diff@1.4.0
1167├── debug@2.0.0 (ms@0.6.2)
1168├── mkdirp@0.5.0 (minimist@0.0.8)
1169├── jade@0.26.3 (commander@0.6.1, mkdirp@0.3.0)
1170└── glob@3.2.3 (inherits@2.0.1, graceful-fs@2.0.3, minimatch@0.2.14)
1172supertest@1.0.1 node_modules/supertest
1173├── methods@1.1.1
1174└── superagent@1.2.0 (extend@1.2.1, methods@1.0.1, cookiejar@2.0.1, component-emitter@1.1.2, reduce-component@1.0.1, mime@1.3.4, qs@2.3.3, debug@2.2.0, formidable@1.0.14, readable-stream@1.0.27-1, form-data@0.2.0)
1176core-js@0.8.4 node_modules/core-js
1178html5@1.0.5 node_modules/html5
1179├── opts@1.2.2
1180└── html5-entities@1.0.0
1182connect-busboy@0.0.2 node_modules/connect-busboy
1183└── busboy@0.2.9 (readable-stream@1.1.13, dicer@0.2.3)
1185request@2.63.0 node_modules/request
1186├── aws-sign2@0.5.0
1187├── forever-agent@0.6.1
1188├── stringstream@0.0.4
1189├── caseless@0.11.0
1190├── oauth-sign@0.8.0
1191├── tunnel-agent@0.4.1
1192├── isstream@0.1.2
1193├── json-stringify-safe@5.0.1
1194├── extend@3.0.0
1195├── qs@5.1.0
1196├── combined-stream@1.0.5 (delayed-stream@1.0.0)
1197├── form-data@1.0.0-rc3 (async@1.4.2)
1198├── tough-cookie@2.0.0
1199├── mime-types@2.1.7 (mime-db@1.19.0)
1200├── http-signature@0.11.0 (assert-plus@0.1.5, asn1@0.1.11, ctype@0.5.3)
1201├── bl@1.0.0 (readable-stream@2.0.2)
1202├── hawk@3.1.0 (cryptiles@2.0.5, sntp@1.0.9, boom@2.9.0, hoek@2.16.3)
1203└── har-validator@1.8.0 (commander@2.8.1, chalk@1.1.1, bluebird@2.10.1, is-my-json-valid@2.12.2)
1205coveralls@2.11.2 node_modules/coveralls
1206├── lcov-parse@0.0.6
1207├── log-driver@1.2.4
1208├── request@2.40.0 (aws-sign2@0.5.0, forever-agent@0.5.2, oauth-sign@0.3.0, stringstream@0.0.4, tunnel-agent@0.4.1, json-stringify-safe@5.0.1, qs@1.0.2, mime-types@1.0.2, form-data@0.1.4, http-signature@0.10.1, hawk@1.1.1, tough-cookie@2.0.0)
1209└── js-yaml@3.0.1 (argparse@0.1.16, esprima@1.0.4)
1211domino@1.0.19 node_modules/domino
1213express-handlebars@2.0.1 node_modules/express-handlebars
1214├── graceful-fs@3.0.8
1215├── promise@6.1.0 (asap@1.0.0)
1216├── object.assign@1.1.1 (object-keys@1.0.7)
1217├── glob@5.0.14 (path-is-absolute@1.0.0, inherits@2.0.1, once@1.3.2, inflight@1.0.4, minimatch@2.0.10)
1218└── handlebars@3.0.3 (source-map@0.1.43, optimist@0.6.1, uglify-js@2.3.6)
1220jshint@2.8.0 node_modules/jshint
1221├── strip-json-comments@1.0.2
1222├── exit@0.1.2
1223├── shelljs@0.3.0
1224├── console-browserify@1.1.0 (date-now@0.1.4)
1225├── minimatch@2.0.8 (brace-expansion@1.1.0)
1226├── htmlparser2@3.8.3 (domelementtype@1.3.0, entities@1.0.0, domhandler@2.3.0, readable-stream@1.1.13, domutils@1.5.1)
1227├── cli@0.6.6 (glob@3.2.11)
1228└── lodash@3.7.0
1230istanbul@0.3.17 node_modules/istanbul
1231├── supports-color@1.3.1
1232├── which@1.0.9
1233├── abbrev@1.0.7
1234├── nopt@3.0.3
1235├── wordwrap@0.0.3
1236├── async@1.3.0
1237├── once@1.3.2 (wrappy@1.0.1)
1238├── esprima@2.4.1
1239├── resolve@1.1.6
1240├── mkdirp@0.5.1 (minimist@0.0.8)
1241├── fileset@0.2.1 (glob@5.0.13, minimatch@2.0.8)
1242├── handlebars@3.0.0 (optimist@0.6.1, source-map@0.1.43, uglify-js@2.3.6)
1243├── escodegen@1.6.1 (esutils@1.1.6, estraverse@1.9.3, optionator@0.5.0, source-map@0.1.43, esprima@1.2.5)
1244└── js-yaml@3.3.1 (esprima@2.2.0, argparse@1.0.2)
1246jscs@2.1.0 node_modules/jscs
1247├── strip-json-comments@1.0.4
1248├── reserved-words@0.1.1
1249├── natural-compare@1.2.2
1250├── babel-jscs@2.0.3
1251├── pathval@0.1.1
1252├── estraverse@4.1.0
1253├── vow@0.4.10
1254├── exit@0.1.2
1255├── commander@2.8.1 (graceful-readlink@1.0.1)
1256├── esprima@2.5.0
1257├── chalk@1.1.0 (escape-string-regexp@1.0.3, supports-color@2.0.0, ansi-styles@2.1.0, has-ansi@2.0.0, strip-ansi@3.0.0)
1258├── lodash.assign@3.2.0 (lodash._baseassign@3.2.0, lodash._createassigner@3.1.1, lodash.keys@3.1.2)
1259├── glob@5.0.14 (path-is-absolute@1.0.0, inherits@2.0.1, inflight@1.0.4, once@1.3.2)
1260├── minimatch@2.0.10 (brace-expansion@1.1.0)
1261├── cli-table@0.3.1 (colors@1.0.3)
1262├── vow-fs@0.3.4 (vow-queue@0.4.2, node-uuid@1.4.3, glob@4.5.3)
1263├── to-single-quotes@1.0.3 (get-stdin@3.0.2, meow@3.3.0)
1264├── to-double-quotes@1.0.1 (get-stdin@3.0.2, meow@3.3.0)
1265├── prompt@0.2.14 (revalidator@0.1.8, pkginfo@0.3.0, read@1.0.6, winston@0.8.3, utile@0.2.1)
1266├── xmlbuilder@2.6.4 (lodash@3.10.1)
1267├── jscs-jsdoc@1.1.0 (comment-parser@0.3.0, jsdoctypeparser@1.2.0)
1268└── babel-core@5.8.22 (slash@1.0.0, shebang-regex@1.0.0, try-resolve@1.0.1, to-fast-properties@1.0.1, trim-right@1.0.1, path-is-absolute@1.0.0, path-exists@1.0.0, fs-readdir-recursive@0.1.2, babel-plugin-remove-console@1.0.1, babel-plugin-remove-debugger@1.0.1, babel-plugin-eval@1.0.1, babel-plugin-inline-environment-variables@1.0.1, babel-plugin-jscript@1.0.4, babel-plugin-property-literals@1.0.1, babel-plugin-member-expression-literals@1.0.1, babel-plugin-react-constant-elements@1.0.3, babel-plugin-undefined-to-void@1.1.6, babel-plugin-react-display-name@1.0.3, babel-plugin-constant-folding@1.0.1, babel-plugin-proto-to-assign@1.0.4, babel-plugin-dead-code-elimination@1.0.2, babel-plugin-runtime@1.0.7, globals@6.4.1, private@0.1.6, esutils@2.0.2, convert-source-map@1.1.1, js-tokens@1.0.1, home-or-tmp@1.0.0, babel-plugin-undeclared-variables-check@1.0.2, line-numbers@0.2.0, debug@2.2.0, repeating@1.1.3, babylon@5.8.22, resolve@1.1.6, source-map@0.4.4, detect-indent@3.0.1, bluebird@2.9.34, json5@0.4.0, output-file-sync@1.1.1, is-integer@1.0.4, source-map-support@0.2.10, lodash@3.10.1, core-js@1.1.0, regexpu@1.2.0, regenerator@0.8.35)
1269[beta-parsoid-update-eqiad] $ /bin/bash -xe /tmp/
1270+ PARSOID_TARGET=/srv/deployment/parsoid
1271+ mkdir -p /srv/deployment/parsoid
1272+ RSYNC_OPTS='--update --times --recursive --links --delete-after --delay-updates'
1273+ rsync --update --times --recursive --links --delete-after --delay-updates /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/parsoid /srv/deployment/parsoid
1274+ rsync --update --times --recursive --links --delete-after --delay-updates /mnt/jenkins-workspace/workspace/beta-parsoid-update-eqiad/deploy /srv/deployment/parsoid
1275[beta-parsoid-update-eqiad] $ /bin/bash -xe /tmp/
1276+ sudo /etc/init.d/parsoid restart
1277Rather than invoking init scripts through /etc/init.d, use the service(8)
1278utility, e.g. service parsoid restart
1280Since the script you are attempting to invoke has been converted to an
1281Upstart job, you may also use the stop(8) and then start(8) utilities,
1282e.g. stop parsoid ; start parsoid. The restart(8) utility is also available.
1283parsoid start/running, process 4058
1284Finished: SUCCESS

On deployment-parsoid05 there is:

curl --silent|python -m json.tool
    "name": "parsoid",
    "sha": "850b02d7f4d55f4d8a2496688b277c6c33a94d3b",
    "version": "0.4.1-git"

Change 243997 merged by jenkins-bot:
parsoid: deploy node_modules with npm

Lets keep the task open for monitoring purpose. If this need to be reverted:

@subbu, @Arlolra -- at SoS release engineering said they are just waiting for us to verify that this change is effective and works well for us. We should run some tests to ensure that beta is being updated as we expect.

Current state:

$ ssh deployment-parsoid05.deployment-prep.eqiad.wmflabs 'curl --silent|python -m json.tool'
    "name": "parsoid",
    "sha": "41f4be5b638dc6f476ca13c29bfc5cd5edff910e",
    "version": "0.4.1-git"

You can see the Jenkins deploy log at . It does not test anything and just /etc/init.d/parsoid restart. Maybe we can add to the job output the curl command above?

Soonish ™ we will migrate the Jenkins job to use scap3 but that is a different story.

After a couple weeks, the Parsoid service on beta cluster seems to run just fine using the source repository and npm install.

Change 243987 merged by Dzahn:
beta: point parsoid back to source code

Change 243992 merged by Dzahn:
beta: parsoid now uses modules defined in source