Page MenuHomePhabricator

wdqs-frontend docker image should (BLUBBER) rebuild automatically when a new patch is pushed to master
Open, NormalPublic5 Story Points

Description

Currently the wdqs-frontend docker image builds from master, but only when build is manually triggered, or a new commit is made to wikibase-docker wmde git repo.

NEW IDEA

Use Blubber...
Do something after using blubber..

OLD IDEA
Instead some jenkins / github hook should hit the travis API / trigger a new build

As this requires a travis token it might be an idea to use a specific account for it that only has access to the wikibase-docker repo under the wmde org.
We have such a user, https://github.com/WikidataBuilder, the question is, who still has the password?

Possible solution
Triggering builds with the API: https://docs.travis-ci.com/user/triggering-build
Webhooks could also work but they would require some extra service to look at the webhooks and determine when the merge actually happened on master...
We could look into a gerrit plugin (as in T87768#3958112)

using the API...

We could create a tiny jenkins job triggering the travis API
The jenkins job could be small and generic, triggering the master build of the given github repo on travis.
Possible ENV vars for the image:

  • AUTH_HEADER (secret token)
  • REPO (example wmde/wikibase-docker)

This will require a travis user token for the requests..

We could also create a small travis job on the GUI repo... https://github.com/wikimedia/wikidata-query-gui triggering the same API request..

Details

Related Gerrit Patches:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Wikidata. · View Herald TranscriptApr 11 2018, 5:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore moved this task from Features to Misc on the Wikibase-Containers board.Apr 25 2018, 2:01 PM
Addshore triaged this task as Normal priority.Jun 26 2018, 3:50 PM

Should just be a call to travis api to build it, or some hook!

Addshore updated the task description. (Show Details)Sep 4 2018, 1:20 PM
Jonas set the point value for this task to 5.Sep 4 2018, 1:25 PM
Jonas moved this task from Ready to estimate to In Progress on the Wikidata-Campsite board.
Addshore updated the task description. (Show Details)Sep 4 2018, 1:28 PM
Addshore updated the task description. (Show Details)Sep 4 2018, 1:37 PM
Addshore updated the task description. (Show Details)
Addshore added a project: wikidata-tech-focus.
Addshore moved this task from Backlog to To Evaluate on the wikidata-tech-focus board.

This didn't get added to the iteration but it should have been

Addshore updated the task description. (Show Details)Sep 10 2018, 8:10 AM
Addshore moved this task from To Evaluate to Ready on the wikidata-tech-focus board.
Addshore updated the task description. (Show Details)Sep 11 2018, 2:51 PM
Addshore added a subscriber: thcipriani.

Claiming this on the campsite.
We are going to invest a little bit of time to see if we could switch the query service Ui to be deployed using Blubber (https://wikitech.wikimedia.org/wiki/Blubber)
This would result in a WMF produced docker image for the service, we could then stop maintaining our own separate image.

@thcipriani just checking, but this would result in an image of whatever is currently on master right?

Restricted Application added a project: User-Addshore. · View Herald TranscriptSep 13 2018, 10:32 AM

@thcipriani just checking, but this would result in an image of whatever is currently on master right?

Depends on how it's setup. For Mathoid a container is built and pushed to docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid that is tagged with a timestamp. This currently happens for every merge to master if the tests are passing and helm test passes.

@thcipriani just checking, but this would result in an image of whatever is currently on master right?

Depends on how it's setup. For Mathoid a container is built and pushed to docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid that is tagged with a timestamp. This currently happens for every merge to master if the tests are passing and helm test passes.

Gotcha, that would work!

Addshore moved this task from Misc to Doing on the Wikibase-Containers board.Sep 19 2018, 10:46 AM

Moving back to Todo as I haven't had time to look at this and other stuff has now come up that is getting some more priority..

Addshore added a comment.EditedNov 8 2018, 1:45 PM

So, while trying to move this to blubber I hit the first issue.

$ docker run --rm -it wdqs-gui-test-$(date --iso)

> wikidata-query-gui@0.1.4 test /srv/service
> grunt test

>> Local Npm module "grunt-auto-install" not found. Is it installed?
>> Local Npm module "grunt-banana-checker" not found. Is it installed?
>> Local Npm module "grunt-contrib-clean" not found. Is it installed?
>> Local Npm module "grunt-contrib-concat" not found. Is it installed?
>> Local Npm module "grunt-contrib-copy" not found. Is it installed?
>> Local Npm module "grunt-contrib-cssmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-htmlmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
>> Local Npm module "grunt-contrib-less" not found. Is it installed?
>> Local Npm module "grunt-contrib-qunit" not found. Is it installed?
>> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
>> Local Npm module "grunt-filerev" not found. Is it installed?
>> Local Npm module "grunt-jscs" not found. Is it installed?
>> Local Npm module "grunt-jsonlint" not found. Is it installed?
>> Local Npm module "grunt-merge-i18n" not found. Is it installed?
>> Local Npm module "grunt-shell" not found. Is it installed?
>> Local Npm module "grunt-stylelint" not found. Is it installed?
>> Local Npm module "grunt-usemin" not found. Is it installed?
Warning: Task "jshint" not found. Use --force to continue.

Aborted due to warnings.
npm ERR! Test failed.  See above for more details.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

It looks like this is due to the dockerfile generated by blubber doing the install of npm packages in one directory:

COPY --chown=65533:65533 ["package.json", "/opt/lib/"]
RUN cd "/opt/lib" && npm install

While having the project files in another dir

WORKDIR "/srv/service
.....
COPY --chown=65533:65533 [".", "."]

and apparently grunts require() method will always look for node_modules directory in the project dir, ignoring the ENV vars??

I guess a related ticket in grunt is https://github.com/gruntjs/grunt/issues/1214

Addshore renamed this task from wdqs-frontend docker image should rebuild automatically when a new patch is pushed to master to wdqs-frontend docker image should rebuild automatically when a new patch is pushed to master.Nov 8 2018, 1:51 PM
Addshore renamed this task from wdqs-frontend docker image should rebuild automatically when a new patch is pushed to master to wdqs-frontend docker image should (BLUBBER) rebuild automatically when a new patch is pushed to master.
Addshore updated the task description. (Show Details)
Addshore moved this task from Backlog to Doing on the Release Pipeline (Blubber) board.

So, while trying to move this to blubber I hit the first issue.

$ docker run --rm -it wdqs-gui-test-$(date --iso)
> wikidata-query-gui@0.1.4 test /srv/service
> grunt test
>> Local Npm module "grunt-auto-install" not found. Is it installed?
>> Local Npm module "grunt-banana-checker" not found. Is it installed?
>> Local Npm module "grunt-contrib-clean" not found. Is it installed?
>> Local Npm module "grunt-contrib-concat" not found. Is it installed?
>> Local Npm module "grunt-contrib-copy" not found. Is it installed?
>> Local Npm module "grunt-contrib-cssmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-htmlmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
>> Local Npm module "grunt-contrib-less" not found. Is it installed?
>> Local Npm module "grunt-contrib-qunit" not found. Is it installed?
>> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
>> Local Npm module "grunt-filerev" not found. Is it installed?
>> Local Npm module "grunt-jscs" not found. Is it installed?
>> Local Npm module "grunt-jsonlint" not found. Is it installed?
>> Local Npm module "grunt-merge-i18n" not found. Is it installed?
>> Local Npm module "grunt-shell" not found. Is it installed?
>> Local Npm module "grunt-stylelint" not found. Is it installed?
>> Local Npm module "grunt-usemin" not found. Is it installed?
Warning: Task "jshint" not found. Use --force to continue.
Aborted due to warnings.
npm ERR! Test failed.  See above for more details.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

It looks like this is due to the dockerfile generated by blubber doing the install of npm packages in one directory:

COPY --chown=65533:65533 ["package.json", "/opt/lib/"]
RUN cd "/opt/lib" && npm install

We've actually changed the way that works in blubber fairly recently. I just updated the releases to ensure that they're all at the latest v0.6.0 like the Debian package used for CI, Let me know if v0.6.0 works for you.

I was on this:

$ blubber --version
0.4.0+9bfa2dd

Will give it a go with this today :)

$ blubber --version
0.6.0+7450643
Addshore added a comment.EditedNov 9 2018, 11:59 AM

So, I gave it another go, and then realized that I wasn't running the docker build with --no-cache.
We might want to work --no-cache into the https://wikitech.wikimedia.org/wiki/Blubber#Example_usage section incase people like me forget.

I then also realized that my build was reading from an out of date Docker file from a previous test that was left in the project directory...
This dockerfile has something to do with the fact that the example command doesn't work on my system(probably a windows thing):

$ blubber ./.pipeline/blubber.yaml test | docker build --no-cache -t wdqs-gui-test-$(date --iso) -f - .
Incorrect function.

So I am using something like this instead:

blubber ./.pipeline/blubber.yaml test > Dockerfile && docker build --no-cache -t wdqs-gui-test-$(date --iso) . && rm Dockerfile

*continues plodding along*

Change 472649 had a related patch set uploaded (by Addshore; owner: Addshore):
[wikidata/query/gui@master] WIP DNM add blubber.yml

https://gerrit.wikimedia.org/r/472649

So I pushed a patch of my work in progress blubber file.

test currently fails as a qunit test fails:

Running "qunit:all" (qunit) task
Testing wikibase/tests/index.html ..........................................................................FF
>> wikibase.queryService.api.CodeSamples - simple
>> Message: could not load code samples
>> Actual: false
>> Expected: true
>> file:///srv/service/wikibase/tests/queryService/api/CodeSamples.test.js:55:17
>> fire@file:///srv/service/node_modules/jquery/dist/jquery.js:3268:36
>> fireWith@file:///srv/service/node_modules/jquery/dist/jquery.js:3398:11
>> file:///srv/service/node_modules/jquery/dist/jquery.js:3622:32

>> wikibase.queryService.api.CodeSamples - empty
>> Message: could not load code samples
>> Actual: false
>> Expected: true
>> file:///srv/service/wikibase/tests/queryService/api/CodeSamples.test.js:55:17
>> fire@file:///srv/service/node_modules/jquery/dist/jquery.js:3268:36
>> fireWith@file:///srv/service/node_modules/jquery/dist/jquery.js:3398:11
>> file:///srv/service/node_modules/jquery/dist/jquery.js:3622:32

Testing wikibase/tests/QueryHelper.html .....................................OK

development currently fails as for some reason some resources are not where they should be?

GET http://localhost:8080/style.css net::ERR_ABORTED 404 (Not Found)
less.js:860 Rendered http://localhost:8080/style.less successfully.
less.js:446 XHR finished loading: GET "http://localhost:8080/style.less".
less.js:860 CSS for http://localhost:8080/style.less generated in 1288ms
less.js:860 Less has finished. CSS generated in 1290ms
less.js:464 [Violation] 'readystatechange' handler took 1017ms
(index):473 GET http://localhost:8080/node_modules/jquery.i18n/libs/CLDRPluralRuleParser/src/CLDRPluralRuleParser.js net::ERR_ABORTED 404 (Not Found)
(index):474 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.js net::ERR_ABORTED 404 (Not Found)
(index):475 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.messagestore.js net::ERR_ABORTED 404 (Not Found)
(index):476 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.fallbacks.js net::ERR_ABORTED 404 (Not Found)
(index):477 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.parser.js net::ERR_ABORTED 404 (Not Found)
(index):478 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.emitter.js net::ERR_ABORTED 404 (Not Found)
(index):479 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.language.js net::ERR_ABORTED 404 (Not Found)
2
less.js:446 XHR finished loading: GET "http://localhost:8080/style.less".
(index):473 GET http://localhost:8080/node_modules/jquery.i18n/libs/CLDRPluralRuleParser/src/CLDRPluralRuleParser.js net::ERR_ABORTED 404 (Not Found)
(index):474 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.js net::ERR_ABORTED 404 (Not Found)
(index):475 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.messagestore.js net::ERR_ABORTED 404 (Not Found)
(index):476 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.fallbacks.js net::ERR_ABORTED 404 (Not Found)
(index):477 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.parser.js net::ERR_ABORTED 404 (Not Found)
(index):478 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.emitter.js net::ERR_ABORTED 404 (Not Found)
(index):479 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.language.js 404 (Not Found)
jquery.js:3600 [Violation] 'setTimeout' handler took 54ms
[Violation] Forced reflow while executing JavaScript took 30ms
jquery.js:3818 jQuery.Deferred exception: Cannot set property 'debug' of undefined TypeError: Cannot set property 'debug' of undefined
    at setLanguage (http://localhost:8080/wikibase/init.js:17:16)
    at HTMLDocument.<anonymous> (http://localhost:8080/wikibase/init.js:38:4)
    at mightThrow (http://localhost:8080/node_modules/jquery/dist/jquery.js:3534:29)
    at process (http://localhost:8080/node_modules/jquery/dist/jquery.js:3602:12) undefined
less.js:464 [Violation] 'readystatechange' handler took 157ms
less.js:446 XHR finished loading: GET "http://localhost:8080/style.less".

I haven't tested prep or production yet.

PS2 @ https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/472649/2 has "test" and "development" fully working.
Now I just have to figure out what to do with "prep" and "production"

Addshore added a subscriber: Joe.Nov 12 2018, 3:04 PM
Addshore changed the task status from Open to Stalled.Nov 12 2018, 3:43 PM

I think this is pretty stalled until the child task regarding an nginx image can be resolved.

Addshore removed Addshore as the assignee of this task.Dec 12 2018, 2:03 PM

Should this stay in the camp iteration then?

Should this stay in the camp iteration then?

I believe so, I think that is the purpose of the stalled column there now.
We still want to get this done as soon as it is unblocked, rather than removing it from the iteration and forgetting about it for months.

Is the nginx base image because you have both a nodejs service listening on a socket in addition to static files that need to be served?

Is the nginx base image because you have both a nodejs service listening on a socket in addition to static files that need to be served?

No, it is currently only static files.
Node is needed to build them, but they can currently be served by any old webserver.

Addshore moved this task from in progress to blocked on others on the Wikidata board.

Removing this from the campsite until it is unblocked.

Change 489386 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[wikidata/query/gui@master] [WIP] [POC] Add blubber and turn it to a nodejs serve-static server

https://gerrit.wikimedia.org/r/489386

I added basic serve-static nodejs libraries that can unblock us from using nginx and it's a good direction to have a nodejs server (then we can inject configs, etc.).
Now I have two issues with prep and deployment:

  • prep needs to have env=prod, that makes npm omit installing lots of needed dependencies like grunt, grunt minifiers, etc.
  • Speaking of grunt, if I remove env=prod from prep step to make sure it works otherwise, it still fails because Gruntfile.js is not copied over before doing the build step:
 ---> Using cache
 ---> b35f86ac2d58
Step 11/34 : ENV APP_BASE_PATH="/srv/service"
 ---> Using cache
 ---> 425e1c70eded
Step 12/34 : COPY --chown=65533:65533 ["package.json", "./"]
 ---> Using cache
 ---> 2cdc1fc68e11
Step 13/34 : RUN npm install
 ---> Running in c065d9733907

(LOTS OF LOGS)

Removing intermediate container c065d9733907
 ---> aa4aed71ea93
Step 14/34 : RUN npm "run" "build"
 ---> Running in 131e5935908e

> wikidata-query-gui@0.1.4 build /srv/service
> grunt build

A valid Gruntfile could not be found. Please see the getting started guide for
more information on how to configure grunt: http://gruntjs.com/getting-started
Fatal error: Unable to find Gruntfile.

npm ERR! Linux 4.15.0-43-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "build"
npm ERR! node v6.12.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! wikidata-query-gui@0.1.4 build: `grunt build`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the wikidata-query-gui@0.1.4 build script 'grunt build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the wikidata-query-gui package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     grunt build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs wikidata-query-gui
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls wikidata-query-gui
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /srv/service/npm-debug.log
The command '/bin/sh -c npm "run" "build"' returned a non-zero code: 1

There are two options now: 1- Move away from Grunt build step (doing it with Make? etc.) it's lots of work 2- Add support for Grunt builds to blubber.
@thcipriani do you think you can do the latter?

Change 489386 abandoned by Ladsgroup:
[WIP] [POC] Add blubber and turn it to a nodejs serve-static server

Reason:
We are not doing this for now

https://gerrit.wikimedia.org/r/489386

Lydia_Pintscher changed the task status from Stalled to Open.Tue, Nov 12, 1:47 PM

Based on story time discussion: Marking this as unstalled because child task was resolved as declined.

I'm in favor of turning the GUI into a nodejs app that serves static files for now and then slowly turning it into a full-fledged nodejs one. @Addshore do you think this is doable?

I'm in favor of turning the GUI into a nodejs app that serves static files for now and then slowly turning it into a full-fledged nodejs one. @Addshore do you think this is doable?

Yes