Page MenuHomePhabricator

Update wikidata-query-gui-build job versions (from Jessie, Node v6, npm v3)
Closed, DeclinedPublic

Description

The wikidata-query-gui-build job, which automatically builds the wikidata/query/gui repository post-merge and submits the result as a new change for the wikidata/query/gui-deploy repository, currently seems to run on Debian Jessie, and uses node v6.11.0 and npm 3.8.3. The npm version is especially concerning to me, since that version predates package-lock.json support (added in v5.0.0) – as a result, it seems to be impossible to reproduce a build locally. It would be great if we could update to a more recent Node+npm, e. g. Node 10 / npm 6.9.0.

Event Timeline

Restricted Application added a project: Wikidata. · View Herald TranscriptOct 16 2019, 1:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Lucas_Werkmeister_WMDE triaged this task as High priority.Oct 31 2019, 11:19 AM

Since I5087b21e83 was merged, our browser tests now require a Node.js with support for async/await syntax, which broke the GUI build.

Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
Addshore added a subscriber: Addshore.

Based on story time today: time-box this to 4 hours.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptTue, Nov 12, 7:35 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.

That was an interesting rabbit hole. The job is actually defined integration/config in jjb/wikidata.yaml and unlike most jobs, it's actually defined in context of a bare metal system or VM and not on a container making it really hard to change, It's pinned in puppet in file modules/contint/manifests/packages/javascript.pp but it has this message:

# DO NOT CHANGE VERSION WITHOUT INVOLVING Krinkle OR hashar
#
# https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup
$versions = {
    'npm'       => '3.8.3',
}

I doubt that we can easily change this version number. @Krinkle or @hashar know better but both are in TechConf right now.

As I said before, we should move to use blubber for wdqs gui instead of this mess.

Ladsgroup removed Ladsgroup as the assignee of this task.Tue, Nov 12, 8:26 PM
Ladsgroup added a subscriber: Ladsgroup.

we should move to use blubber for wdqs gui instead of this mess

Yes we should.
This also aligns with what we want to do with the docker images for wikibase next year (having them all built using the pipeline), but for the wmde wikibase docker images they would probably be in their own namespace in the registry.
So for if we would use the pipeline for building images of the UI for use in production, I guess we need to discuss that in T235639 T238303 and T192006
I'm going to write ticket for the wmde wikibase docker image side of this very soon, as for the prod side, I guess we need to talk to the search platform team and ops / service ops.

For now / until then, maybe we should just manually build and push them?
We can still deploy.
Interested to know @Lucas_Werkmeister_WMDE thoughts here, as I'm pretty sure he is normally the one making this build commit (at least he has for the xss issues over the past weeks.

Well, I only started doing it because the automated builds broke due to this task. Anyone should be able to do the same thing by running grunt deploy (depending on your Gerrit/SSH config, you might need to do the final git push from the build/ directory by hand).

I don’t have a lot of thoughts about how it should work instead, but I can at least say that anything Docker-based is not going to help my productivity, seeing as Docker still doesn’t support cgroupsv2.

I don’t have a lot of thoughts about how it should work instead, but I can at least say that anything Docker-based is not going to help my productivity, seeing as Docker still doesn’t support cgroupsv2.

Sorry I should have been more clear, moving to docker is going to help developer productivity by streamlining the deployment process and reducing the lead time. I don't know exactly how that would affect the work of developing using docker. Why not supporting cgroupsv2 would be a problem for you? Can you elaborate?

It means I can’t run Docker locally, because my system uses cgroupsv2, so I can’t develop using Docker.

It means I can’t run Docker locally, because my system uses cgroupsv2, so I can’t develop using Docker.

Moving the deployment and build for production to docker and the deployment pipeline would not require a local dev to run docker in order to develop on the UI. npm start etc would still exist and work etc.

WMDE-leszek added a subscriber: WMDE-leszek.EditedTue, Dec 3, 9:53 AM

The Docker discussion seems unrelated in the sense that we might want bump the npm version used separately from moving the UI (including the whole disentanglement of UI and backend game) to the deployment pipeline, right?

As the outcome of the investigation: what would be the simplest way to make wikidata query gui go up to npm 6 or so?
Edit: to ask the obvious: we are talking about node/npm version used in CI jobs/build jobs, right?

# DO NOT CHANGE VERSION WITHOUT INVOLVING Krinkle OR hashar
#
# https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup
$versions = {
    'npm'       => '3.8.3',
}

I doubt that we can easily change this version number. @Krinkle or @hashar know better but both are in TechConf right now.

I guess we need to re ping these 2 lovely people to find out!

Ping @Krinkle @hashar

hashar added a comment.Tue, Dec 3, 1:09 PM

Since I5087b21e83 was merged, our browser tests now require a Node.js with support for async/await syntax, which broke the GUI build.

The CI job runs on Jessie CI instances which are obsolete. Being Jessie they come with Node v6.11.0 (and npm 3.8.3).

We are not going to add any Stretch or Buster instances for this purpose, the job has to migrate to a Docker T210286: Migrate wikidata-query-gui-build to Docker containers. I haven't looked into that migration since there is a sub task to migrate to Blubber / release pipeline: T192006: wdqs-frontend docker image should (BLUBBER) rebuild automatically when a new patch is pushed to master.

thanks @hashar for the thorough explanation. I somehow imagined there is already a Docker-powered counterpart for this job. Not doing migration separately from the blubber migration makes total sense.
I guess this task could be declined, as realistically there will be not much done around it?

Addshore closed this task as Declined.Tue, Dec 3, 4:57 PM

Will mark as decline as I believe this is covered by all of the other tasks that it is linked to.