Page MenuHomePhabricator
Create Task
Maniphest T284125

cxserver CI fails errors from unrelated project
Closed, ResolvedPublic
Actions

Description

Log:

> cxserver@1.2.0 test /srv/service
> npm run lint && npm run unittest


> cxserver@1.2.0 lint /srv/service
> eslint .


/srv/service/function-schemata/javascript/src/util.js
  4:22  error  "yaml" is not found  node/no-missing-require

/srv/service/function-schemata/javascript/src/utils.js
  5:22  error  "yaml" is not found  node/no-missing-require

Full log example: https://integration.wikimedia.org/ci/job/service-pipeline-test/9039/console
Pipeline view https://integration.wikimedia.org/ci/blue/organizations/jenkins/service-pipeline-test/detail/service-pipeline-test/9039/pipeline

(the build has been flagged to be kept for the purpose of this task)

Here function-schemata is not related to cxserver at all.

Details

SubjectRepoBranchLines +/-
Revert "Make sure workspace is clean on checkout"integration/pipelinelibmaster+2 -2
Customize query in gerrit

Related Objects

Event Timeline

KartikMistry created this task.Jun 2 2021, 6:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2021, 6:38 AM
• Nikerabbit subscribed.Jun 2 2021, 7:52 AM
hashar updated the task description. (Show Details)Jun 2 2021, 12:40 PM
KartikMistry added a comment.Jun 2 2021, 12:56 PM

CI looks good now!

hashar edited projects, added Release Pipeline; removed Continuous-Integration-Config.Jun 2 2021, 1:15 PM

The file /srv/service/function-schemata/javascript/src/utils.js would come from the repository mediawiki/services/function-schemata. The only thing I can imagine is that the container got build reusing some cached layer and that erroneously used a layer that was intended for function-schemata :-\

If I got it right the test container was run with:

docker run --name 'plib-run-xzpffatv' sha256:'99529c90bda3ac4962e87e204bf1b19f648e0f9bdc54ea395447feb4b8f52e68'

When looking at the image, under /srv/services there are cxserver files timestamped on June 2 at 05:51 and beside those:

$ ls -la|grep -v 05:51
total 440
drwxr-xr-x   1 root     root       4096 May 15 03:46 ..
drwxr-xr-x   7 somebody somebody   4096 Jun  2 00:02 function-schemata
drwxr-xr-x 582 somebody somebody  20480 May 25 03:31 node_modules
-rw-r--r--   1 somebody somebody     48 Jun  1 20:49 targets.yaml

And somehow .git/config has a few submodules:

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = git://contint2001.wikimedia.org/mediawiki/services/cxserver
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "prod"]
	active = true
	url = https://gerrit.wikimedia.org/r/wikimedia/portals/deploy
[submodule "function-schemata"]
	active = true
	url = https://gerrit.wikimedia.org/r/mediawiki/services/function-schemata
[submodule "modules/translators"]
	active = true
	url = https://github.com/zotero/translators
[submodule "modules/zotero"]
	active = true
	url = https://github.com/zotero/zotero.git

That apparently comes from the Jenkins job workspace which has:

/srv/jenkins-slave/workspace/service-pipeline-test
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = git://contint1001.wikimedia.org/mediawiki/services/cxserver
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "prod"]
	active = true
	url = https://gerrit.wikimedia.org/r/wikimedia/portals/deploy
[submodule "function-schemata"]
	active = true
	url = https://gerrit.wikimedia.org/r/mediawiki/services/function-schemata
[submodule "modules/translators"]
	active = true
	url = https://github.com/zotero/translators
[submodule "modules/zotero"]
	active = true
	url = https://github.com/zotero/zotero.git

Which is suspicious.

Looking at the image history:

sudo docker history 99529c90bda3ac4962e87e204bf1b19f648e0f9bdc54ea395447feb4b8f52e68 
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
99529c90bda3        7 hours ago         /bin/sh -c #(nop)  LABEL zuul.commit=5094828…   0B                  
645b16c68457        7 hours ago         /bin/sh -c #(nop)  LABEL jenkins.job=service…   0B                  
730146a41d7c        7 hours ago         /bin/sh -c #(nop)  LABEL jenkins.build=9039     0B                  
ed1f9ef733c9        7 hours ago         /bin/sh -c #(nop)  LABEL blubber.variant=tes…   0B                  
7212b5c301b8        7 hours ago         /bin/sh -c #(nop)  ENTRYPOINT ["npm" "test"]    0B                  
ec22fd9e6805        7 hours ago         /bin/sh -c #(nop)  ENV NODE_ENV=                0B                  
bcc82238d808        7 hours ago         /bin/sh -c #(nop)  ENV HOME=/home/runuser       0B                  
12409ca5488a        7 hours ago         /bin/sh -c #(nop)  USER 900                     0B                  
7e7a219f0af0        7 hours ago         /bin/sh -c #(nop) COPY --chown=65533:65533di…   209MB               
7871c725c08e        8 days ago          /bin/sh -c npm install                          205MB               
f815c6380de6        2 weeks ago         /bin/sh -c #(nop) COPY --chown=65533:65533fi…   1.81kB              
dbf556a55c13        2 weeks ago         /bin/sh -c #(nop)  ENV APP_BASE_PATH=/srv/se…   0B                  
416298d96a21        2 weeks ago         /bin/sh -c #(nop) WORKDIR /srv/service          0B                  
5434b4d62925        2 weeks ago         /bin/sh -c #(nop)  ENV HOME=/home/somebody      0B                  
6edc100d7568        2 weeks ago         /bin/sh -c #(nop)  USER 65533                   0B                  
56c4e041eb8c        2 weeks ago         /bin/sh -c (getent group "900" || groupadd -…   9.17kB              
051382ccc952        2 weeks ago         /bin/sh -c (getent group "65533" || groupadd…   8.99kB              
2360c8b548d7        8 weeks ago         /bin/sh -c apt-get update && apt-get install…   249MB               
afd1fc979732        2 months ago        /bin/sh -c #(nop)  ENV DEBIAN_FRONTEND=nonin…   0B                  
3b7900c2515f        2 months ago        /bin/sh -c #(nop)  ENV HOME=/root               0B                  
6508a7aa7811        2 months ago        /bin/sh -c #(nop)  USER 0                       0B                  
bf379a1552f5        22 months ago       |4 HTTPS_PROXY=http://webproxy.eqiad.wmnet:8…   78MB                
<missing>           22 months ago       /bin/sh -c #(nop) COPY file:ed374e0a9cfac84b…   3.13kB              
<missing>           2 years ago                                                         56.2MB              Imported from -

I examined those images and function-schemata got introduced in the image 7e7a219f0af0. Which does a COPY of the workspace. And surely that is how the files landed in. In that image they have the timestamp Jun 1 20:49

I have saved the build workspace to /srv/jenkins-slave/workspace/service-pipeline-test-T284125 . It does not contain the function-schemata directory. So that is an entire mystery.

KartikMistry added a comment.Jun 3 2021, 6:28 AM

And, it is failing again: https://integration.wikimedia.org/ci/job/service-pipeline-test/9046/console

hashar added subscribers: jeena, • dancy, dduvall.Jun 3 2021, 9:44 AM

Yeah that will keep continuing but I can not find how extra modules get added to .gitmodules apparently from different repositories. The workspace does have a .git/modules with some stalled content:

$ ls -la .git/modules
total 20
drwxr-xr-x 5 jenkins-slave jenkins-slave 4096 Apr 28 09:08 .
drwxr-xr-x 9 jenkins-slave jenkins-slave 4096 Jun  3 09:00 ..
drwxr-xr-x 8 jenkins-slave jenkins-slave 4096 Jun  3 05:50 function-schemata
drwxr-xr-x 4 jenkins-slave jenkins-slave 4096 Apr 28 09:08 modules
drwxr-xr-x 8 jenkins-slave jenkins-slave 4096 Jun  1 14:08 prod

$ ls -la .git/modules/modules
total 16
drwxr-xr-x 4 jenkins-slave jenkins-slave 4096 Apr 28 09:08 .
drwxr-xr-x 5 jenkins-slave jenkins-slave 4096 Apr 28 09:08 ..
drwxr-xr-x 8 jenkins-slave jenkins-slave 4096 May 13 12:15 translators
drwxr-xr-x 9 jenkins-slave jenkins-slave 4096 May 13 12:15 zotero

Maybe the submodules keep accumulating cause the workspace does not get cleaned and some magic would detect them and add them back to .gitconfig? I don't know really.

I think the issue lies in jjb/service-pipeline.groovy which might not wipe the workspace. It starts with:

stage('Checkout patch') {
  checkout(patchset.getSCM())
}

Which would mean we need some extra option in integration/pipelinelib file src/org/wikimedia/integration/PatchSet.groovy. It does use the Git plugin clean before checkout:

[$class: 'CleanBeforeCheckout', deleteUntrackedNestedRepositories: true],

That got introduced recently by @jeena with https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/676141 and replaced the WipeWorkspace option previously used. I guess clean does not reclaim untracked repositories under .git/modules and they end up being "helpfully" added back to .gitmodules.

gerritbot added a comment.Jun 3 2021, 6:54 PM

Change 697825 had a related patch set uploaded (by Jeena Huneidi; author: Jeena Huneidi):

[integration/pipelinelib@master] Revert "Make sure workspace is clean on checkout"

https://gerrit.wikimedia.org/r/697825

gerritbot added a project: Patch-For-Review.Jun 3 2021, 6:54 PM

Change 697825 merged by jenkins-bot:

[integration/pipelinelib@master] Revert "Make sure workspace is clean on checkout"

https://gerrit.wikimedia.org/r/697825

Stashbot added a comment.Jun 3 2021, 7:06 PM

Mentioned in SAL (#wikimedia-releng) [2021-06-03T19:06:32Z] <hashar> contint1001 and contint2001: deleted all workspaces under /srv/jenkins-slave/workspace/* # T284125

hashar closed this task as Resolved.Jun 3 2021, 7:08 PM
hashar claimed this task.

That is fixed for real now. We might have published image that come with the extra git modules though, though I haven't looked at the other jobs workspaces to confirm whether it would have happen or not. Does not seem to big a bit concern to me anyway.

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2021, 7:10 PM
KartikMistry added a comment.Jun 4 2021, 4:38 AM

Thanks @jeena and @hashar!

KartikMistry triaged this task as Medium priority.Jun 4 2021, 5:45 AM
KartikMistry added a project: Language-Team (Language-2021-April-June).
KartikMistry moved this task from Quarter Backlog to Done on the Language-Team (Language-2021-April-June) board.
hashar mentioned this in T290608: Pipeline lib still leaks containers on contint1001 / contint2001.Sep 8 2021, 6:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits