Page MenuHomePhabricator

wdqs-frontend docker image should (BLUBBER) rebuild automatically when a new patch is pushed to master
Closed, DeclinedPublic5 Estimated Story Points

Description

Currently the wdqs-frontend docker image builds from master, but only when build is manually triggered, or a new commit is made to wikibase-docker wmde git repo.

NEW IDEA

Use Blubber...
Do something after using blubber..

OLD IDEA
Instead some jenkins / github hook should hit the travis API / trigger a new build

As this requires a travis token it might be an idea to use a specific account for it that only has access to the wikibase-docker repo under the wmde org.
We have such a user, https://github.com/WikidataBuilder, the question is, who still has the password?

Possible solution
Triggering builds with the API: https://docs.travis-ci.com/user/triggering-build
Webhooks could also work but they would require some extra service to look at the webhooks and determine when the merge actually happened on master...
We could look into a gerrit plugin (as in T87768#3958112)

using the API...

We could create a tiny jenkins job triggering the travis API
The jenkins job could be small and generic, triggering the master build of the given github repo on travis.
Possible ENV vars for the image:

  • AUTH_HEADER (secret token)
  • REPO (example wmde/wikibase-docker)

This will require a travis user token for the requests..

We could also create a small travis job on the GUI repo... https://github.com/wikimedia/wikidata-query-gui triggering the same API request..

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@thcipriani just checking, but this would result in an image of whatever is currently on master right?

Depends on how it's setup. For Mathoid a container is built and pushed to docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid that is tagged with a timestamp. This currently happens for every merge to master if the tests are passing and helm test passes.

@thcipriani just checking, but this would result in an image of whatever is currently on master right?

Depends on how it's setup. For Mathoid a container is built and pushed to docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid that is tagged with a timestamp. This currently happens for every merge to master if the tests are passing and helm test passes.

Gotcha, that would work!

Moving back to Todo as I haven't had time to look at this and other stuff has now come up that is getting some more priority..

So, while trying to move this to blubber I hit the first issue.

$ docker run --rm -it wdqs-gui-test-$(date --iso)

> wikidata-query-gui@0.1.4 test /srv/service
> grunt test

>> Local Npm module "grunt-auto-install" not found. Is it installed?
>> Local Npm module "grunt-banana-checker" not found. Is it installed?
>> Local Npm module "grunt-contrib-clean" not found. Is it installed?
>> Local Npm module "grunt-contrib-concat" not found. Is it installed?
>> Local Npm module "grunt-contrib-copy" not found. Is it installed?
>> Local Npm module "grunt-contrib-cssmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-htmlmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
>> Local Npm module "grunt-contrib-less" not found. Is it installed?
>> Local Npm module "grunt-contrib-qunit" not found. Is it installed?
>> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
>> Local Npm module "grunt-filerev" not found. Is it installed?
>> Local Npm module "grunt-jscs" not found. Is it installed?
>> Local Npm module "grunt-jsonlint" not found. Is it installed?
>> Local Npm module "grunt-merge-i18n" not found. Is it installed?
>> Local Npm module "grunt-shell" not found. Is it installed?
>> Local Npm module "grunt-stylelint" not found. Is it installed?
>> Local Npm module "grunt-usemin" not found. Is it installed?
Warning: Task "jshint" not found. Use --force to continue.

Aborted due to warnings.
npm ERR! Test failed.  See above for more details.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

It looks like this is due to the dockerfile generated by blubber doing the install of npm packages in one directory:

COPY --chown=65533:65533 ["package.json", "/opt/lib/"]
RUN cd "/opt/lib" && npm install

While having the project files in another dir

WORKDIR "/srv/service
.....
COPY --chown=65533:65533 [".", "."]

and apparently grunts require() method will always look for node_modules directory in the project dir, ignoring the ENV vars??

I guess a related ticket in grunt is https://github.com/gruntjs/grunt/issues/1214

Addshore renamed this task from wdqs-frontend docker image should rebuild automatically when a new patch is pushed to master to wdqs-frontend docker image should rebuild automatically when a new patch is pushed to master.Nov 8 2018, 1:51 PM
Addshore renamed this task from wdqs-frontend docker image should rebuild automatically when a new patch is pushed to master to wdqs-frontend docker image should (BLUBBER) rebuild automatically when a new patch is pushed to master.
Addshore updated the task description. (Show Details)
Addshore moved this task from Backlog to Doing on the Release Pipeline (Blubber) board.

So, while trying to move this to blubber I hit the first issue.

$ docker run --rm -it wdqs-gui-test-$(date --iso)

> wikidata-query-gui@0.1.4 test /srv/service
> grunt test

>> Local Npm module "grunt-auto-install" not found. Is it installed?
>> Local Npm module "grunt-banana-checker" not found. Is it installed?
>> Local Npm module "grunt-contrib-clean" not found. Is it installed?
>> Local Npm module "grunt-contrib-concat" not found. Is it installed?
>> Local Npm module "grunt-contrib-copy" not found. Is it installed?
>> Local Npm module "grunt-contrib-cssmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-htmlmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
>> Local Npm module "grunt-contrib-less" not found. Is it installed?
>> Local Npm module "grunt-contrib-qunit" not found. Is it installed?
>> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
>> Local Npm module "grunt-filerev" not found. Is it installed?
>> Local Npm module "grunt-jscs" not found. Is it installed?
>> Local Npm module "grunt-jsonlint" not found. Is it installed?
>> Local Npm module "grunt-merge-i18n" not found. Is it installed?
>> Local Npm module "grunt-shell" not found. Is it installed?
>> Local Npm module "grunt-stylelint" not found. Is it installed?
>> Local Npm module "grunt-usemin" not found. Is it installed?
Warning: Task "jshint" not found. Use --force to continue.

Aborted due to warnings.
npm ERR! Test failed.  See above for more details.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

It looks like this is due to the dockerfile generated by blubber doing the install of npm packages in one directory:

COPY --chown=65533:65533 ["package.json", "/opt/lib/"]
RUN cd "/opt/lib" && npm install

We've actually changed the way that works in blubber fairly recently. I just updated the releases to ensure that they're all at the latest v0.6.0 like the Debian package used for CI, Let me know if v0.6.0 works for you.

I was on this:

$ blubber --version
0.4.0+9bfa2dd

Will give it a go with this today :)

$ blubber --version
0.6.0+7450643

So, I gave it another go, and then realized that I wasn't running the docker build with --no-cache.
We might want to work --no-cache into the https://wikitech.wikimedia.org/wiki/Blubber#Example_usage section incase people like me forget.

I then also realized that my build was reading from an out of date Docker file from a previous test that was left in the project directory...
This dockerfile has something to do with the fact that the example command doesn't work on my system(probably a windows thing):

$ blubber ./.pipeline/blubber.yaml test | docker build --no-cache -t wdqs-gui-test-$(date --iso) -f - .
Incorrect function.

So I am using something like this instead:

blubber ./.pipeline/blubber.yaml test > Dockerfile && docker build --no-cache -t wdqs-gui-test-$(date --iso) . && rm Dockerfile

*continues plodding along*

Change 472649 had a related patch set uploaded (by Addshore; owner: Addshore):
[wikidata/query/gui@master] WIP DNM add blubber.yml

https://gerrit.wikimedia.org/r/472649

So I pushed a patch of my work in progress blubber file.

test currently fails as a qunit test fails:

Running "qunit:all" (qunit) task
Testing wikibase/tests/index.html ..........................................................................FF
>> wikibase.queryService.api.CodeSamples - simple
>> Message: could not load code samples
>> Actual: false
>> Expected: true
>> file:///srv/service/wikibase/tests/queryService/api/CodeSamples.test.js:55:17
>> fire@file:///srv/service/node_modules/jquery/dist/jquery.js:3268:36
>> fireWith@file:///srv/service/node_modules/jquery/dist/jquery.js:3398:11
>> file:///srv/service/node_modules/jquery/dist/jquery.js:3622:32

>> wikibase.queryService.api.CodeSamples - empty
>> Message: could not load code samples
>> Actual: false
>> Expected: true
>> file:///srv/service/wikibase/tests/queryService/api/CodeSamples.test.js:55:17
>> fire@file:///srv/service/node_modules/jquery/dist/jquery.js:3268:36
>> fireWith@file:///srv/service/node_modules/jquery/dist/jquery.js:3398:11
>> file:///srv/service/node_modules/jquery/dist/jquery.js:3622:32

Testing wikibase/tests/QueryHelper.html .....................................OK

development currently fails as for some reason some resources are not where they should be?

GET http://localhost:8080/style.css net::ERR_ABORTED 404 (Not Found)
less.js:860 Rendered http://localhost:8080/style.less successfully.
less.js:446 XHR finished loading: GET "http://localhost:8080/style.less".
less.js:860 CSS for http://localhost:8080/style.less generated in 1288ms
less.js:860 Less has finished. CSS generated in 1290ms
less.js:464 [Violation] 'readystatechange' handler took 1017ms
(index):473 GET http://localhost:8080/node_modules/jquery.i18n/libs/CLDRPluralRuleParser/src/CLDRPluralRuleParser.js net::ERR_ABORTED 404 (Not Found)
(index):474 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.js net::ERR_ABORTED 404 (Not Found)
(index):475 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.messagestore.js net::ERR_ABORTED 404 (Not Found)
(index):476 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.fallbacks.js net::ERR_ABORTED 404 (Not Found)
(index):477 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.parser.js net::ERR_ABORTED 404 (Not Found)
(index):478 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.emitter.js net::ERR_ABORTED 404 (Not Found)
(index):479 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.language.js net::ERR_ABORTED 404 (Not Found)
2
less.js:446 XHR finished loading: GET "http://localhost:8080/style.less".
(index):473 GET http://localhost:8080/node_modules/jquery.i18n/libs/CLDRPluralRuleParser/src/CLDRPluralRuleParser.js net::ERR_ABORTED 404 (Not Found)
(index):474 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.js net::ERR_ABORTED 404 (Not Found)
(index):475 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.messagestore.js net::ERR_ABORTED 404 (Not Found)
(index):476 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.fallbacks.js net::ERR_ABORTED 404 (Not Found)
(index):477 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.parser.js net::ERR_ABORTED 404 (Not Found)
(index):478 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.emitter.js net::ERR_ABORTED 404 (Not Found)
(index):479 GET http://localhost:8080/node_modules/jquery.i18n/src/jquery.i18n.language.js 404 (Not Found)
jquery.js:3600 [Violation] 'setTimeout' handler took 54ms
[Violation] Forced reflow while executing JavaScript took 30ms
jquery.js:3818 jQuery.Deferred exception: Cannot set property 'debug' of undefined TypeError: Cannot set property 'debug' of undefined
    at setLanguage (http://localhost:8080/wikibase/init.js:17:16)
    at HTMLDocument.<anonymous> (http://localhost:8080/wikibase/init.js:38:4)
    at mightThrow (http://localhost:8080/node_modules/jquery/dist/jquery.js:3534:29)
    at process (http://localhost:8080/node_modules/jquery/dist/jquery.js:3602:12) undefined
less.js:464 [Violation] 'readystatechange' handler took 157ms
less.js:446 XHR finished loading: GET "http://localhost:8080/style.less".

I haven't tested prep or production yet.

PS2 @ https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/472649/2 has "test" and "development" fully working.
Now I just have to figure out what to do with "prep" and "production"

Addshore changed the task status from Open to Stalled.Nov 12 2018, 3:43 PM

I think this is pretty stalled until the child task regarding an nginx image can be resolved.

Should this stay in the camp iteration then?

Should this stay in the camp iteration then?

I believe so, I think that is the purpose of the stalled column there now.
We still want to get this done as soon as it is unblocked, rather than removing it from the iteration and forgetting about it for months.

Is the nginx base image because you have both a nodejs service listening on a socket in addition to static files that need to be served?

Is the nginx base image because you have both a nodejs service listening on a socket in addition to static files that need to be served?

No, it is currently only static files.
Node is needed to build them, but they can currently be served by any old webserver.

Change 489386 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[wikidata/query/gui@master] [WIP] [POC] Add blubber and turn it to a nodejs serve-static server

https://gerrit.wikimedia.org/r/489386

I added basic serve-static nodejs libraries that can unblock us from using nginx and it's a good direction to have a nodejs server (then we can inject configs, etc.).
Now I have two issues with prep and deployment:

  • prep needs to have env=prod, that makes npm omit installing lots of needed dependencies like grunt, grunt minifiers, etc.
  • Speaking of grunt, if I remove env=prod from prep step to make sure it works otherwise, it still fails because Gruntfile.js is not copied over before doing the build step:
 ---> Using cache
 ---> b35f86ac2d58
Step 11/34 : ENV APP_BASE_PATH="/srv/service"
 ---> Using cache
 ---> 425e1c70eded
Step 12/34 : COPY --chown=65533:65533 ["package.json", "./"]
 ---> Using cache
 ---> 2cdc1fc68e11
Step 13/34 : RUN npm install
 ---> Running in c065d9733907

(LOTS OF LOGS)

Removing intermediate container c065d9733907
 ---> aa4aed71ea93
Step 14/34 : RUN npm "run" "build"
 ---> Running in 131e5935908e

> wikidata-query-gui@0.1.4 build /srv/service
> grunt build

A valid Gruntfile could not be found. Please see the getting started guide for
more information on how to configure grunt: http://gruntjs.com/getting-started
Fatal error: Unable to find Gruntfile.

npm ERR! Linux 4.15.0-43-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "build"
npm ERR! node v6.12.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! wikidata-query-gui@0.1.4 build: `grunt build`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the wikidata-query-gui@0.1.4 build script 'grunt build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the wikidata-query-gui package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     grunt build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs wikidata-query-gui
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls wikidata-query-gui
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /srv/service/npm-debug.log
The command '/bin/sh -c npm "run" "build"' returned a non-zero code: 1

There are two options now: 1- Move away from Grunt build step (doing it with Make? etc.) it's lots of work 2- Add support for Grunt builds to blubber.
@thcipriani do you think you can do the latter?

Change 489386 abandoned by Ladsgroup:
[WIP] [POC] Add blubber and turn it to a nodejs serve-static server

Reason:
We are not doing this for now

https://gerrit.wikimedia.org/r/489386

Lydia_Pintscher changed the task status from Stalled to Open.Nov 12 2019, 1:47 PM

Based on story time discussion: Marking this as unstalled because child task was resolved as declined.

I'm in favor of turning the GUI into a nodejs app that serves static files for now and then slowly turning it into a full-fledged nodejs one. @Addshore do you think this is doable?

I'm in favor of turning the GUI into a nodejs app that serves static files for now and then slowly turning it into a full-fledged nodejs one. @Addshore do you think this is doable?

Yes

Addshore changed the task status from Open to Stalled.EditedJan 16 2020, 11:36 PM

This ticket relates to the wmde supported ui docker image.
Marking this as stalled until the work in T241291 is done, the we can see how to move forward.

Change 472649 abandoned by Addshore:
WIP DNM add blubber.yml

https://gerrit.wikimedia.org/r/472649

Although this is currently in the backlog in a prioritized position it may be best tackled as a trailblaze rather than on campsite as pointed out by @WMDE-leszek
Thus if the story time gets to this ticket perhaps this should be evaluated and potentially skipped for the next one in the backlog?

Awaiting the task to be opened by @Addshore describing the current state of the next steps to take in this field

Moving out of the backlog as this will be rewritten and proposed as T264710: Host static sites on kubernetes and we need some answers there before we can continue

I want to point out that we don't necessary need nginx or any static file serving software in kubernetes to be able to move this. The GUI can be easily turned to a nodejs app or worst case, you can use https://www.npmjs.com/package/http-server to serve them.

don't necessary need nginx or any static file serving software
The GUI can be easily turned to a nodejs app

Using my complete lack of involvement for a light-hearted comment about the irony of this discussion.
Instead of managing the risk of an accepted technology that will enabled many we make teams individually resort to workarounds opening up more and potentially bigger problems in the process.

I'm going to decline this task for now.
The release strategy hike may end up deciding to do in this direction, but the reasons this ticket was created originally are far gone now.
T235639: Decouple Query Service UI deployment from Query Service deployment was merged into T241291: Simplify WDQS Packaging which is also now done.
The next step is further decoupling this deployment per T264710#6586070 and we can move on from there in the future.